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1 .) 


Introduction 


The following constitutes an Interim report on NASA Contract 
NASW-3350, " A Study of Software Management and Guidelines for Flight 
Projects." The contract calls for a iurvey of present software develop* 
ment policies and practices - Task #1; an analysis of these policies 
’ oractices - Task #2; and a final report recommending possible Im- 
provements - Task #3. This report summarizes the Task #1 and Task #2 
efforts to-date, essentially providing the background Information 
necessary to assess the adequacy of present NASA flight software develop- 
ment approaches. Further analysis and refinement of this intot. n-, 
partially as a result of a Workshop involving key NASA software practi- 
tioners, is expected to form the basis upon which preliminary Improve- 
ment recommendations can be developed for NASA review. 

2. ) Purp ose 

The motivating NASA goal for this study was. to enhance the timely 
and cost-effective production of reliable flight software. The immediate 
objectives of this study, in support of that goal, are to refine NASA- 
wide understanding of flight software development successes and problems 
and, as a result, develop management policy Improvement recommendations 
that can be broadly agreed to within the NASA, particularly at the Center 
Directors' and practitioners’ levels. 

3. ) Scop e 

The software upon which this study is focused is described below 
(a slightly modified version of the present NMI 2410.6 Scope). 

"These requirements shall apply to all of the software for a 
selected flight program- -whether it is onboard software, support 
software, ground software necessary to perform the mission opera- 
tions or software used to conduct the major system level testlng-- 
that could, in any way, cause mission failure; seriously degrade the 
mission objectives; or adversely affect personnel safety. 

Ground software that is institutional and multiuser, or part of a 
generic capability shall be identified as to its use in the Program, 
where it is documented, and how it Is managed, but it Is not con- 
trolled by this NMI. However, all mission peculiar or mission unique 
changes which must be made to Institutional software to support a 
flight mission shall fall within the scope of these instructions." 
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The recommendations that eventually will result from the survey 
and analysis tasks reported on herein will emphasize NASA Headquarters- 
level, policy-oriented suggestions (eg., those that might Improve the 
2410.6). However, additional recommendations will address the 
broader class of technical, educational, and managerial Issues, Including 
■•“ntlflcatlon of potential follow-on efforts. 

iu farther characterize the scope and focus of this study. It may 
be useful to contrast those subjects that typically will be addressed 
with those that, at least within this initial effort, will not be ad- 
dressed. 

o Subjects that will not be addressed : 

- Requirements or design specification methodologies or tools 

- Coding or testing techniques, tools, or facilities 

- Modeling or simulation technology or facilities 

- Programming languages, operating system technology, or support 
software 

o Subjects that will be addressed : 

- Policy for management of software development — what to do, not 
how to do it. 

- Impact of the abstract and complex nature of software. 

- Alleged inadequacy of management policies, procedures, planning, 
standards, control, focus, visibility, etc. 

- Evolutionary nature of requirements. Iterative nature of process, 
and contradictory nature of demands. 


Approach 

As implied by the previously stated objectives, the present study is 
intended to be a fact-finding effort. While certain of the resulting 
recommendations may be of immediate benefit, it is anticipated that identi- 
fication of potentially fruitful avenues for future efforts will be 
equally important. 

Several study assumptions seamed inescapable from the outset. For 
instance, a significant degree of operational autonomy for each of the NASA 

Centers was assumed to be desirable, if not unavoidable. Therefore, partici- 
pation of major NASA software implementation Centers in this study was 
considered mandatory. Dialogue amongst these key NASA elements was con- 
sidered to be essential if the management policy recommendations that 
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result are to be effective.... either In the view of managers who must 
enforce the policies or developers who must abide by them. 

The role of the C.S. Draper Laboratory Is seen primarily, therefore, 
as that of a non-aligned organizing agent. Intermediary, catalyst, and 
reporter. 

4t the direction of NASA Headquarters, a group of NASA flight software 
Centers/projects were selected as study-subjects (JSC/Shuttle, MSFC/Space 
Telescope, JPL/Galileo, GSFC/Landsat D). In addition, another Government 
agency (Air Force - ESD) and two Industry representatives (TRW and SDC) 
were Included to capture outside experience. Many additional sources 
could be Identified but this would, It was felt, unacceptably limit the 
degree of analysis each received. Section A-2 of the Appendix lists the 
Interviewees and prospective Workshop participants. 

The order of events for this study were as follows: 

o Preliminary review of each study-subject's policy documentation 
....to Initiate understanding of present policies and to prepare 
for the survey interviews to follow. A checklist of topical 
areas of potential importance to the survey was developed to 
serve as a common baseline. (See appendix, A-4). 

o Study-subject interviews. .. .to solicit personal experiences and 
suggestions, and to obtain answers to generic and study-subject- 
specific questions. An interview format was generated to help 
focus these sessions. (See appendix, A-3). 

o Interim P.eport preparation. .. .refining and documenting the In- 
formation obtained from the above reviews and Interviews. In 
addition, a draft listing of generic lessons-leamed has been ex- 
tracted including successful .practices, potential problem areas/ 
deficiencies, useful observations, potential recomrendatlons, etc. 

We are here now! 

o Workshop/Seminar with the multiple objectives of acquiring 

concise identifi cation, by experts, of key software development 
deficiencies and potential corrective measures, and providing 
them the opportunity for informal dialogue. (Scheduled for 
May 29th and 30th at MIT's Endicott House). 

o Final Report preparation — converting the draft observa- 
tions and Workshop experience into specific recommendations for 
review by a NASA peer group prior to publication. 
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Participation by C.S. Draper Laboratory Individuals on the Space 
Telescope. Galileo, and Landsat D software audit teams last year provided 
Insight into the valuable "evaluation" aspects of the present VMl 2410.6 
audit process. This study avoids duplication of those program-specific 
•>di t activities In order to en ourage candor during the survey of present 
•»s. The organization of the Interim and Final reports avoids corre- 
lation or identified problem areas with specific study-subjects. Rather, 
a separate Final Report listing of generic problem areas will complement 
the non-„udgemental , study-subject specific characterizations of this report. 

As a final note, concern has been raised on several occasions tiu.. 
policy recommendations generally draw too heavily op lessons learned from 
the past, overlooking the demands anticipated In the future and leading, 

In our case, to policies for the 70's rather than the 80's. We believe 
our approach addresses this issue constructively and It Is a key Item In 
the draft list of preliminary observations. In addition, we plan to dis- 
cuss the Issue at the Workshop. 
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5. ) Study-Subject Characterizations 


The Survey and Analysis tasks of this Study Included a review of 
selected study-subject policy documentation and follow-up interviews with 
' •'v study-subject software managers and practitioners. 

T he following sections characterize each study-subjects' software 
development process by briefly describing their present policies, procedures, 
responsibility and authority assignments, review and control mechanisms, 
documentation, standards, etc. The appendix (section A-l ) contains an 
informal collection of more detailed background information on each :-ludy- 
subject in the varying styles of each reviewer. 

It should be noted that these descriptions are limited in both 
context and completeness. It would be misleading to draw comparative con- 
clusions based on them due to the varying emphasis of each reviewer. We 
solicit expansion, correction, and clarification, particularly by those 
study-subject personnel who have not yet had the opportunity to review the 
material . 

However, our primary thrust, especially during the interviews, has 
been on determining which policies and practices the study-subjects felt 
were or would be useful, NASA-wide, rather than on detailed explanations 
of their present operations. Therefore, the non-judgement! , study-subject 
specific content of this Interim Report serves primarily as background 
material for the generic, (i.e. non study-subject specific) judgemental 
Final Report recommendations. 

The Study-Subject characterizations that follow are listed below: 

5.1 JSC/Shuttle 

5.2 MSFC/Space Telescope 

5.3 GSFC/landsat D 

5.4 JPL/Galileo 

5. 5 TRW 

5.6 SDC 

5.7 Air Force - ESD 
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JSC/Space Shuttle 

The software development process for the Space Shuttle 
Program is quite accurately represented by the policies docu- 
mented in JSC 07700, Volume XVIII, Book .3, Software Manage- 
ment and Control, Revision 9, dated December 19, 1978. A 
ner ic model of the software development life-cycle Is 
described within the framework of a system development process. 

The life-cycle is divided into phases: requirements definition, 

preliminary design, detailed design, coding, verification and 
integrated testing. Each phase (except coding) is followed by 
a project (e.g. Orbiter) level review. The required activities 
of each review are specified including which review-item(s) are 
to be placed under NASA configuration control. The final review 
results in end-item acceptance. The policy also calls for 
reviews at the software developers (e.g. JSC/SSD-IBM) level to 
review schedules, cost and performance for each Important pro- 
duct and activity. These latter reviews are to be specified in 
the software development plan. 

A three level configuration control process is described. 

At the program level, the Program Requirements Control Board (PRCB) 
controls the top-level computer system and software requirements 
(JSC 07700, Vol . XVIII, Books 1-3), the Master Verification Plan 
and the inter-project (e.g., Orbiter-SSME) systems -software inter- 
faces. The Space Shuttle Program Manager is the Chairman of the 
PRCB. At the project (Orbiter) level the Configuration Control 
Board (CCB) has the authority and responsibility to establish 
baselines for each software deliverable end-item; to approve imple- 
mentation standards (e.g., programming languages) and test 
acceptance criteria; and to control changes, performance 
levels, costs, risks, and schedules. The chairman of the orbiter 
CCB is the Director of the Orbiter Project office. The Orbiter 
CCB has delegated this responsibility to the Orbiter Avionics 
Software Control Board (OASCB) for the flight software. The 
chairman of the OASCB is the Director of the Orbiter Avionics 
Project office. The software developer (JSC/SSD-IBM) Is responsi- 
ble for supporting the project level configuration control process 
and for performing the detailed configuration management of the 
software designs, builds, tests, and related documentation. 



Book 3 describes the configuration control process and a flow 
chart Is Included that Illustrates the order of activities at 
all three levels (program, project, and software developer). 

The generic types and minimum contents of the documentation 
required to support software development and verification Is 

ified. There are twelve such documents Identified and grouped 
Into five categories: 

1) Management: Management Plan, Development Plan, Quality 
Assurance Plan, 

2) Performance and Design Requirements: Requirements Speci- 

fication, 

3) Design: Preliminary Design Specification, Detailed Design 

Specification, Programmer's Handbook, 

4) Test Planning: Test Requirements, Test Plan, and 

5) Testing: Development Test Specification, Software System 

Test Specification, Verification Test Report. 

Books 1 and 2, while less pertinent to this study, are 
characterized below. 

Book 1 of Volume XVIII, ALLOCATION OF COMPUTATIONAL FUNCTIONS, 
delineates the top-level computational responsibilities of each 
major computational element (Orbiter, Launch and Landing, Mission 
Operations, Main Engine) during the various operational phases 
(Installation and Checkout, Mate and Interface, Preflight, Flight, 
Post-Landing, and Flight Planning). 

Book 2, ALLOCATION OF SIMULATION FUNCTIONS, defines the roles 
of major Shuttle simulators and allocates functions to them. 

Finally, Book 3 has a section that generically describes the 
precursory system level information and documents that are required 
by the software development activity to design, develop and verify 
the software. 



MSFC/Space Telescope 

Two MSFC software management policy documents were the 
subjects of this study. They are: 1) MSFC SOFTWARE MANAGEMENT 
REQUIREMENTS FOR FLIGHT PROJECTS, MMI and 2) MSFC SOFTWARE 
'4NAGEMENT AND DEVELOPMENT REQUIREMENTS, MA-001-006-2H. 

Thu first Is a brief (6 pages) top-level policy document 
that Is in part identical to portions of NMI 2410.6. 

The second is comprised of seven books that describe re- 
quirements In specific terms. The subject matter (titles) of 
the seven books are: 


1. 

Overview and Guidelines 


2. 

Software Configuration Control 


3. 

Documentation, Deliverables, and 

Planning Data 

4. 

Software Data Requirements 


5. 

Software Standards, Procedures, 

and Quality Assurance 

6. 

Parent Systems Development 


7. 

Indeoendent Verification and Validation 


Neither of these documents is project specific. All 
policies and requirements are stated in generic terms and each pro- 
ject is free to adapt them to their specific environment provided 
they comply with the intent of the policies. 

Like the NMI, the MMI requires (unless specifically waived) 
that all software used on complex, high-profile flight projects 
comply to these policies. The MMI also outlines the authorities 
of the offices and groups that have primary responsibility for 
management of software development. Of particular note: the 

procurement office is charged with insuring that policy provisions 
are included In all applicable contracts. The responsibility of 
a Peer Review Committee is also Included. 



The Seven-book MA describes the activities required In the 
software development from thi conceptual phase through operations and 
maintenance. Book 1 briefly describes each phase: conceptual, 
requirements, design, code and debug, verification, validation, 
systems Integration, and operations and maintenance. Guidelines 
for the degree of applicability of the policies are Included In 
Book 1. The degree of applicability is based on the size, com- 
plexity and criticality of the project. Guidelines for uu. 
ware reviews to be conducted at key development milestones are 
also Included In Book 1. These reviews are: systems requirements 
review (SRR), systems design review (SDR), software requirements 
review (SRR), software preliminary design review (PDR), software 
critical design review (CDR), software test review (TR), functional 
configuration Inspection (FCI), configuration inspection (Cl), and 
acceptance (AR). These software review milestones are placed 
within the context of the parent system's development cycle. 

Book 2 describes the software configuration change control 
process, Including a flow diagram that depicts the required 
sequence of activities. The various forms used to document changes 
and discrepancies are also included and described. The responsi- 
bilities of three levels of control autnority are outlined: the 

Program/Project Configuration Control Board (CCB), the £ rf ware 
Review Board (SRB), and the software developer's internal control. 

Book 3 describes the Documentation, Deliverables and Planning 
Data for MSFC projects developing software. Twenty- four documents 
covering management, development, test, and operations are described 
and a documentation tree showing their interrelationship Is included. 
Guidelines for the number and physical form of typical software 
deliverables are Included. Six types of software planning data 
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are described: Software Function Description, Software Specifica- 
tion Document Tree, List of Computer Program Contract End Items 
(CPCEI's), Development Schedules, Support Facilities, and Management 
Summaries. 

Book 4 contains a Data Requirements (DR) form for each of 
the twenty-four documents described in Book 3. The DR contains sub- 
mittal requirements and a description of the document. 

Book 5 describes the standards, procedures and quality 
assurance requirements to be used during software development. The 
standards and procedures cover topics such as: structured programming. 
Internal i .tware data and hardware tests, naming standards, input- 
output interfaces. Programmer Notebook, coding, and testing. The 
responsibility of the software developer's QA program Is described. 
These responsibilities cover such areas as configuration management, 
testing, deficiency correction, and documentation The QA function 
is required to be independent of lower level technical management. 

Book 6 is a high level description of a typical parent system 
development process from conception through integration and acceptance. 
The relationship between the systems and software development Mfe 
cycles is bri fly described. 

Book 7 describes a three phase Independent Verification and 
Validation (IV&V) process. Each phase consists of two parallel 
activities: Phase 1 covers reouirements analysis and test planning. 

Phase ? covers equation and design analysis and facility development, 
and Phase 3 covers coding analysis and testing. The IV&V documentation 
requirements and management review responsibilities are briefly 
described. 
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5.3 GSFC/Landsat-D 


The Goddard Space Flight Center (GSFC) personnel inter- 
•iewed were involved in three distinctly different flight soft- 
v. -elated efforts. They were: 

1) Flight software produced by an industry contractor 
as part of a larger spacecraft contract (Landsat-D) ; 

2) Flight software produced internally by GSFC p«__. . el; 

3) Flight supportive ground software produced primarily 
by "captive" contractor personnel under GSFC techni- 
cal supervision. 

The management approach of each of these was quite differ- 
ent. Each of the categories will be discussed individually. 

All of the documentation reviewed as part of this study was used 
or generated in support of item ”1)" above, the Landsat-D program. 

Discussion 

1. The flight software for Landsat-D is beinq procured as 
part of a larger contract for the spacecraft and integration. 

The type of contract chosen was a function of system requirements 
and not a function of the requirements for software, hardware, 
integration, etc. individually. 

Subsequent to contract award, negotiations took place to 
insure the adequacy of the management plans and their conform- 
ance to GSFC requirements. It has been stated that NMI 2410.6 
considerations will need to be included in future RFPs to insure 
proposals are in compliance. 
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The General Electric SEAM document is the model upon which 
the Landsat-D software management is based. Figure 1 shows the 
SEAM flight segment software development process and is extracted 
from the SEAM document. The SEAM document defines this process 
~neric terms to enable application to a broad range of pro- 
jects. The SEAM consists of four levels? 

1) Policy - Top level statement of policy and goals; 

2) General Instruction - Mandates software c " ’ ' •'°’*lng 
and management policies using a top-level repre- . nta- 
tion of the process model and a brief elaboration of 
xtems in the model; 

3) Standard Outline - Provides skeleton outlines defining 
content of all documents supportive of mandated poli- 
cies, and 

4) Guidelines - Examples, annotated outlines and guides 
to support 2) and 3) above. 

interaction between GSFC and the contractor for both techni- 
cal and management concerns occur at frequent intervals (on the 
order of every other week) and at periodic milestone reviews. 

The more frequent meetings are based upon the evolution of docu- 
ments that support milestone reviews such as the "Requirements 
Baseline" and management concerns such as schedule and personnel 
availability. These meetings in general are involved with in- 
scope contract issues. The milestone meetings are a culmination 
of the previous meetings and involves management in the approval 
of technical documents and resolution of contract issues. 

The contractor responsible for development is also respon- 
sible for operations and maintenance over the first 90 days of 
the spacecraft mission. A separate contract will be let for 
subsequent operations by open competition. 

A review of the Landsat-D documentation indicates most of 
the concepts in NMI 2410.6 are addressed. 
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SOFTWARE ENGINEERING AND 
MANAGEMENT PROCESS MODEL. 



Figure 1. Top-Level Representation of the SEAM Process 
Model for Software Development Projects 









2. GSFC in-house software development has been accomplished 
for programs such as the "Orbiting Astronomical Observatory (OAO) " 
and the "International Ultraviolet Explorer (IUE)". The size of 
the software task has grown with each successive program and while 

''anagement controls and constraints on software development have 
bee?, he most part informal, an effort has been made to tailor 
the management policies to fit the task at hand. The OAO program was 
small enough to allow a close informal working relationship between 
programmers, providing free flow of technical infoi.»at.L*. , little 
management visibility. The software requirements/specification 
document grew as the software was produced and had no formal controls. 
The IUE program which followed was somewhat larger and more complex 
and justified a more formal management atmosphere. A formal require- 
ments document was generated prior to the start of coding and was the 
responsibility of the systems engineer. The document was controlled 
by a "mini" configuration control ioard. This approach provided 
visibility and opportunity for management control . 

It is understood that a means of implementing NMI 2410.6 in 
the GSFC internal environment is in process. 

3. The flight supportive ground software discussed has been 
used for non-real-time applications (several minutes delay) such 
as spacecraft attitude determination and determination of maneuver 
and flight control information. The software was produced by con- 
tractor personnel, procured under a level-of-ef fort contract, under 
the technical supervision of GSFC personnel. 

The contractor personnel had imposed upon them a management 
policy derived by the segment of GSFC responsible for the software 
being produced. The policies are documented; however, the imposi- 
tion of the policies is by mutual agreement and not by contract. 

In this case the policies have evolved during the period over 
which the same contractor has been used. 

The present NMI 2410.6 does not apply to the software develop- 
ment; however, the responsible GSFC personnel feel that most of the 
concepts in NMI 2410.6 are addressed in their policies. 



5.4 Jet Propulsion Laboratory/Gal i.1 eo 


The Project Galileo Software Management 
Plan, Volume I, Technical Development and 
Management Approach (1979), is up to date and 
matches the actual software development process 
as described during our interview at JPL. 

Software development is made difficu.... ' u«e 
it originates from 4 sources: 1) new software 

developed for Galileo, 2) software inherited 
from Voyager, 3) Mission Control and Computing 
Center (MCCC) Software with its own policies, 
and pre-existing software, and 4) Deep Space 
Network (DSN) Software with its own policies 
and pre-existing software. Chapter 5 of the 
Plan specifically addresses the exceptions to 
the Plan necessary for realistic development of 
a project in the JPL environment with this pre- 
existing software. Figure 1-1 shows the Plan 
Organization Overview with its seven life-cycle 
phases: Requirements Development, Preliminary 

Design, Detailed Design, Coding and Debugging, 
Testing, Integration and Delivery, and Maintenance. 
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Figure 1-1. Plan Organization Overview 








Thirteen Milestones/Technical Reviews corresponding 

to these phases are detailed in that Plan, along with 

documents and responsible parties. Four of these 

« 

Milestones specifically call for a schedule/cost 
analysis to update either the System for Resource 
Man^ement of the Project Office or one of the Technical 
Division Software Implementation Plans. 

During the Requirements Development phase the three 
Project System Functional Requirements Documents 
(Navigation, Orbiter, and Mission Operations) are 
divided by the System Engineers into Subsystems 
Functional Requirements with a corresponding Software 
Functional Description Document. The Integrated 
Software Functional Diagram is then generated. With that, 
the Cognizant Engineers generate Preliminary then Final 
Requirements Documents (SRD), User's Guide (UG), and 
Interface Specs (SIS). Next, while Cognizant Programmers 
generate the General Design Document in PDL for xhe 
Preliminary Design Review, in parallel the Cognizant 
Engineers generate the Acceptance Test Plan. 

The General Design Document (GDD) is expanded for 
the Critical Design Review by the programmers, and a Phase II 
User's Guide and Interface Spec are completed. Coding 
proceeds by the Programmer in a top down manner and ends 
with unit testing. Three further levels of testing are 
specified. Acceptance testing by the Cognizant Engineer 
demonstrates the software to be a stand-alone product. 
Integrated Testing by the IV&V contractor demonstrates 
the software as an element of the subsystem and of the 
system. The last testing phase consists of end-to-end 
scenarios in a pseudo-realistic operational environment. 
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When the software is then transferred to 
Maintenance Phase, each component has an identified 
Sustaining Engineer and a Regression Test Baseline - 
one or two small test cases used in acceptance testing and 
available to check modified software or check proposed 
system level changes. 

The Project Configuration Management Pin*' 

(PD625-32) defines four product baselines: Requirem^.. : ■ 

(Built-To), Design (Code-To), Program (As-Built), and 
Regression Baseline (As-Built Compliance). A formal 
change control procedure is defined for software 
categorized as mission critical under direction of the 
Project Software System Manager, (or higher), via a 
Software Change Control Board. For other software, a 
local change control procedure is available to the 
Project Software System Engineer. A third change 
procedure is available to the Software System Engineer 
for emergencies in urgent situations. 

The Plan additionally covers Flight Software 
Reprogrammability, Margin Management, Quality 
Assurance, and Portability. 

Volume II of the Plan, Software Technical and 
Administrative Procedures (STAPs), is quite comprehensive. 
STAPs are accumulated in separate sections, usually 
just prior to each new step in the development process, 
under the responsibility of the Project Software System 
Engineer. STAPs cover detailed standards and implemen- 
tation procedures and provide for a "common interpre- 
tation" of the plan. 
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The Deep Space Net (DSN) was also reviewed. 

Its historic practices are well described in Standard 
Practices for the Implementation of Computer Software 
(1978). The current revision, JPL Standard Practice, 
Implementation and Management of JPL Software (Draft- 
April 15, 1980, Preliminary), has the additional feature 
that management policies and detail required i. 
documentation products are different for each of four 
classes of software: 1) Critical to Mission success 

(or very expensive, or very large, or many users), 

2) Significant to JPL task performance, 3) Important 
for organization effectiveness, and 4) Important for 
individual effectiveness (and inexpensive, and small, 
and used only by developer). 
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This section (5.5) is proprietary 
to TRW and requires their 
permission before distribution 


I 
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5.5 TRW 


The System Engineering and Integration Division 
(SEID) Software Development Policies (1977-1978) 
und the SEID Software Products Standards (1977-1978) 
are the policy and standards from which a project 
management plcn is derived. Not unexpectedly, they 
closely match Air Force policies of the same tir 
frame and, in fact, reference the MIL standard documents 
in which they are in compliance. A simplified over- 
view of the Life Cycle and associated documents is 
attached as Figure 2-1. (The grand three-dimensional 
TRW lifecycle reproduces best in wall poster size). 

Eight development phases are identified: 
Conceptual, Definition. Preliminary Design, Detailed 
Design, Development, Unit Test, Integration Test , and 
Acceptance Test. 

The System Spec is generated during the Conceptual 
Phase. From that, the Definition Phase generates the 
Software Requirements Spec, the Software End Product 
Acceptance Plan, the Interface Spec, Data Requirements 
Plan, QA plan, and Configuration Management Plan for 

Review at the Software Requirements Review (SRR). 

The Preliminary and Detailed Design Phases follow 
with the Preliminary and Critical Design Reviews 
(PDR and CDR) covering the Data Base Specification, 
the Design Spec, the Unit Test Plan, the Integration 
Test Plan, the Software Acceptance Test Plan, and 
the User's Manual. The Standards document contaLns 
examples of the required documentation and identifies 
which parts are mandatory. 


21 




Figure 2-1. Time Phasing of SEIO Software Document at ? on 
Set During Project Life Cycle 













Unit Development Folders are the key management 
tool of the Development Phase. Procedures specify 
Top Down development, independent design check via 
c design walk-through, and guidelines for good program- 
ming. The three testing phases execute the three 
required test plans (unit, integration, and acceptance) 
and culminate with the Functional Conf igurati . r » idit. 
and the Physical Conf iguration Audit. 

The functions of Configuration Management are 
specified in the Configuration Management Manual. 
Requirements for four basic functions are defined: 

1) Identification, 2) Configuration Control 
Mechanisms, 3) Status Accounting, and 4) Config- 
uration Verification. The degree of formality and 
control employed and manpower used is conditioned 
by the size and complexity of the project, the signifi- 
cance of the project, and the investment risks. 

The Policy also specifies ongoing review and 
periodic quality audits that are to follow a Quality 
Assurance Plan. The Division Product Assurance 
Manager (PAM) is identified as responsible for the 
quality assurance functions and the configuration 
management functions. The QA Division is organised 
as a separate Division at TRW. 



This section (5.6) Is proprietary 
to the System Development Corp. 
and requires their permission 
before distribution. 


24 



5.6 System Development Corporation 


In January of 1966, the System Development Corporation (SDC) issued a 
software Development Manual (SDM), one of three components of their Software 
haciuij approach to software development . The other components are a set 
of computer-based tools, and an organizational philosophy and discipline. 

The SDM contains a statement of corporate- approved F v ' **ftware 

development, and a phase-by-phase description of software products and 
activities. 

The six SDC software development phases are: 

• Planning 

• Requirements/Performance 

e Design 

• Development 

» T cst and Acceptance 

• Operations and Maintenance 

In addition, the SDM describes phase independent activities: project and 

contract management, configuration management, and quality assurance. The 
basic documents produced during software development include a project plan, 
a computer program (CP) performance specification (requirements), a CP design 
specification (preliminary design), a CP detailed development specification 
(detailed design), a programmer's reference manual, a version description 
document, and test and user documentation. Besides the usual design reviews, 
the SDM calls for separate explicit reviews of the project plan, and the CP 
performance specification. 


The SDM stresses the use of top-down design and development. Implementa- 
tion by successive builds, and reliance on a program production library for 
configuration management. 

Our discussions at SDC confirmed that the SDM is a reasonably up-to-date 
-'t.ion of actual practices. The manual Is used more for reference than 
for training. It is also useful in describing their system to management 
within SDC, and to potential customers. 
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5.7 Air Force. Electronic Systems Division 


Between 1975 and 1978* the Air Force Electronic Systems Division 
(Air Force Systems Command) contracted for the development of a Software 
''cquisition Management Guidebook (SAMG) series for command, control and 
cu.-m. . -atlons (C 3 ) systems. Additional guidebook series are being 
developed for flight simulator and avionics software. The C series con- 
sists of 16 volumes covering such topics as documer *ation , verification, 
validation, life cycle events, contracting, maintenance, and software co. 
estimation (See Appendix 1.7 for a. complete list). They were written by the 
MITRE Corporation and the System Development Corporation. 

The intended audience for the series is Air Force program office manage- 
ment personnel who are responsible for managing software acquisition. The 
information presented includes material to supplement the official standards 
documents (lessons learned, common mistakes, etc.), checklists and descrip- 
tions of proven techniques, and references to the official directives and 
regulations pertaining to the topic under discussion. 

The practices described are rooted in the DOD system acquisition life 
cycle which has a well-defined set of phases and milestones (reviews and 
audits). The basic phases are (see Figure 1): 

• conceptual 

• validation 

• full-scale development 

• production and deployment 

Essential to the process are a set of specification doc ...tents, first at 
the system level, then at the configuration item or component level. 
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SYSTEM ACQUISITION CYCLE 
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bunmary of System Acquisition, Model CPCI, Com 
Program Life Cycle, and Command Responsibility 


Computer programs have been designated configuration items within the frame- 
work, and component specification documents (requirements and design) have 
been defined to govern their acquisition and configuration management. 

These *nd other documents are reviewed and authenticated at various formal 

♦•ones, such as the preliminary and critical design reviews (PDR and CHR), 

The guidebook series presents a coherent picture of this evolving body 
of standards and practice. 
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6. ) Summary 


Each of the study-subjects has what appears to be effective software 
management and development practices in place. Though they differ slightly 
In style, form, and level of specificity, they all address roughly the same 
r of concern and follow generally the same life-cycle patterns. 

io..^ of these differences, particularly between NASA Centers that 
often must work together on software projects, could perhaps, constructively 
be eliminated or clarified. However, these variations of terminology, style, 
form, etc, reflect the differing organizations and "ways or o-.i r ”'' c s" 
that goes along with the high degree of (and evidently desirable) Center 
autonomy. This issue, ie, desirable degrees of Software Management Practices 
commonality within NASA, will be discussed at the Workshop and In the Final 
Report. 


i 


i 
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APPENDIX 
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A-l Expanded Study-Subject Characterizations 

This background information has been gathered from documentation 
reviews and study-subject Interviews. It Is written In the individual styles 
of the various study team reviewers, though the enclosed Interview Format 
^ and Checklist (A-4) formed a common baseline of sorts. 

Iho Study-Subject Expanded Characterizations that follow are listed 

below: 

A-l.l OSC/Shuttle 

A-l. 2 MSFC/Space Telescope 

A-l. 3 GSFC/Landsat D 

A-l. 4 JPl /Galileo 

A- 1.5 TRW 

A-l. 6 SDC 

A-l .7 Ai r Force - ESD 
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A-l.l Characterization of JSC Shuttle Software Management and Control Policy . 


The Space Shuttle Program Level Requirements are documented 
JSC 07700. Volume XVIII of this nineteen volume set is the Com- 
pute! Jystem and Software Requirements. The writing of Volume XVIII 
was undertaken in late 1972 by the Data Systems Analysis Directorate 
of JSC in response to a request by the Program Director. Book 3 
of Volume XVIII, SOFTWARE MANAGEMENT AND CONTROL, is trie 
subject of this effort, as It defines the program level policy for 
Shuttle Software Development. Book 3 has been revised ten times 
since official publication in 1974. Revision 9, dated December 19, 
1978, was used in this study (revision 10 is a small editorial 
correction) . 

As previously noted, JSC 07700 is a program level document and 
is intended to apply to all Shuttle software elements, however this 
study was restricted to the policy as it applied to the Orbiter 
prime system flight software. The requirement to adhere to this 
policy has been included in the software development contract with 
IBM since the inception of the contract. The program has an active 
policy to maintain JSC 07700 as evidenced by the fact that ten page 
changes to Book 3 have been issued since its initial publication. 

An examination of Book 3 vis-a-vis tht checklist of this study 
revealed substantia 1 agreement. The policy adequately covers most of 
the categories of the checklist. The same is true of the policy vis-a- 
vis the contents of NMI 2410.6. The characterization of this document 
follows the categurical elements of the checklist. 


1 . Top Level Policy 

JSC 07700, Volume XVIII, Book 3 is the Software Manage- 
ment and Control policy document at the Space Shuttle Program 
Manager level. All elements of the Space Shuttle Program 
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(NASA and contractor orgain 2 ations) must adhere to the policy. 
Book 3 describes the generic software development process, 
the configuration control process, and specifies the control 
responsibilities, documentation, and review milestones. 

The following software reviews are required. 

1) The Software Design Requirements Review (DKk) ir. he’:' 
prior to Initiating formal preliminary software design. 

2) The Software Preliminary Design Review (PDR) is held to 
review the preliminary (functional) design before 
detailed design proceeds. 

3) The Critical Design Review (CDR) is held to review the 
CODE-TO design, test acceptance criteria, and to place 
the detailed design under configuration control. 

4) The Configuration Inspection (Cl) is held to perform 
the following: 

Review changes from CGDE-TO to AS-BUILT software 
design 

Review compliance of AS-BUILT software to require- 
ments 

Review verification test results 

Place AS-BUILT software under configuration control 

Release software for systems and integrated tests 
(note: pre-CI test releases are permitted by 

waiver) . 
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5) The Acceptance Review (AR) Is neld at the systems level 

^ to perform the following: 

. Review the software changes since Cl and related 
test results 

Review the system and integrated acceptance tests 
Establish acceptability of the software. 

6) Incremental Reviews can be held when deemed appropriate 
in order to place portions of the software and documenta- 
tion under configuration control. 

The policy specifies the end-purpose of each review a. '•clim- 
ates the documents and products to be reviewed. It also specifies 
the minimal contents of a Review Summary Report that must be published 
immediately after each review. A figure that illustrates the major 
phases, elements and milestones of the software development process 
is included. 

2. Software Management Plan 

JSC 07700 establishes the requirement for a Software 
1 Management Plan to be produced early in the software develop- 

ment cycle. 

Plan objective: establish specific management policies and 

define the means by which these policies will be implemented. 
Plan shall : 

. Define specific configuration control plan and the pro- 
cess(es) to be followed for defining the procedures for 
baseline definition, design review, review item disposi- 
tion (RID), change control and change evaluation and 
reporting 

Define approval authority for software milestone reviews 
(DRR, PDR, CDR, Cl) 

Establish the charter and responsibility of all organi- 
zations, boards, panels and working groups, including 
functions and interfaces of each plus functions associated 
with management of each. A work breakdown structure (WBS) 
will be included 

| Provide an overview of facilities required and define 

responsibilities for facility planning 


35 


. Define specific documents required for software develop- 
ment and responsibilities for preparation of same 
. Establish management policies and responsibilities for 
ICO development, baselining, technical coordination, 
and review 

Define roles and responsibilities for estab'lshment and 
control of all Q.A. requirements 

Define status reporting procedures to provide visibility 
of progress, including problem identification and dlsposl 
tion. 

3. Con figuration Management 

JSC 07700 contains an excellent, extensive section 
(16 pages) on Configuration Management. All terms used are 
generic and well defined. A generic model (flow diagram) of 
the configuration control process is included. The organiza- 
tions responsible for the various levels and stages of con- 
figuration control are identified. Three levels are identified 
Program, Project and Software Developer's Internal Control. 

The stages identified are the DRR, PDR, CDR, Cl and AR. The 
documents that are subject to configuration control are identi- 
fied. 

This section contains a requirement (section 3.6) that 
the software developer's organization shall have a Quality 
Assurance function that is no_t subject to direction from 
lower level technical management (which it shall audit). 


4. 


JSC 07700 cites the requirement for a Software Quality 
Assurance Plan (section 4.2.3) The purpose of the plan is 
to ensure that uniform QA requirements are Imposed during 
software development and test. The plan shall: Establish 

procedures for fulfilling all software QA requirements, define 
the QA review and reporting procedures, and establish auditing 
procedures for 1) design, coding and verification standards, 

2) test procedure, and 3) tape/card handling. 
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5. End-Product Acceptance 

JSC 07700 describes a two-stage acceptance of the soft- 
ware. The first stage is approval of the software for release 
for systems and Integration testing. This is performed at 
the Configuration Inspection (Cl) based on the review of soft- 
ware verification test results and the AS-BUILT Detailed Soft- 
ware Design Specification (see section 3.2.4). The first stage 
can be augmented with incremental reviews (see section 3.2.6). 
The Second stage is the Acceptance Review (AR). This encom- 
passes a review of the software performance during in* .-a ted 
systems testing plus a review of all software changes and 
corresponding verification test results since Cl (see section 
3.2.5). 

JSC 07700, Vol , XVIII, Book 3, Software Management and 
Control (the book under review) states that the Software 
Mangement Plan shall define the approval authority and review 
procedures for the Cl. 

The AR is a system level review and Is performed in 
accordance with JSC 07700, Vol. IV. 

6 . Operation & Maintenance Plan 

JSC 07700 does not refer to such a plan. Operations is 
covered by: 1) a User's Guide, sect. 4. 4. 2. 2 (part of the 

DSDS), 2) the System Operating Manual; sect 5.7.1 and 3) POD 
generates a Mission Operations Plan for each missicn/mlssion- 
type. 

Maintenance is perhaps best covered by the configuration 
control requirements outlined in the Software Management Plan 
(sect. 4.2.1) and in the Software Developer's Internal (Con- 
figuration) Control (section 3.3.3). 

7 . Resource & Sche d ule Estimation, Monitoring and Control . 

This topic is addressed In the Software Management and 
Development Plan Sections. 

The Management Plan must define the status reporting 
procedures necessary to provide visibility of progress, in- 
cluding problem identification and disposition. It must also 


37 


define control and approval responsibilities. 

The Development plan must contain the schedules, activi- 
ties and resources required for software development. 

8. Risk Assessment 

The project level CCB has the responsibil ity tc minimize 
development risks, but there is no specific focus on the ways 
or means to assess risks. 

9. Documentation 

JSC 07700 describes twelve development documents. These 
documents are typed as to the level of NASA configuration con- 
trol. Type 1 documents must be approved by the project level 
CCB prior to implementation. Type 2 documents do not require 
fcrmal NASA review and approval, but must be reviewed and 
approved by the organization responsible for generating the 
document. The twelve documents (more fully described else- 
ware in this report) are: 

Software Management Plan 
Software Development Plan 
. Software Quality Assurance Plan 

Software Requirements Specification 
. Preliminary Software Design Specif i cat '.on 
Detailed Software Design Specification 
Programmer's Handbook 
Softwrre Test Requirements 
Software Test Plan 

Software Development Test Specification 
Software System Test Specification 
Software Verification Test Report 

All are typ^ 1 except the Programmer's Handbook, which 
is type 2 . 

10. Development Plan 

JSC 07700 calls for a Software Development Plan the ob- 
jective of which is to present the schedules, activities, re- 
sources, and milestones required for the development of the 
so f twa re . 
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11 . 


Requirements Specifications 


JSC 07700 specifies the existence of three types of per- 
formance requirements specification documents. 

System Requirements Specification 

Purpose: Define functional , performance, and inter- 
face design requirements for each system. 

Will include 

. traceability of each requirement to source 
. will address all system functional characteris- 

tics 

. Preliminary computer characteristics 

System performance requirements in measurable/ 
quantified terms with established acceptance 
criteria for each 

Identify operational requirements that impact 
hardware/ software design 

Key assumptions and constraints in defining the 
external system interface requirements 
Systems quality assurance requirements. 

Inter-Project Interface Control Documents (ICDs) 

Purpose: Identify and control all interfaces that 

will affect the computer system and software design 
or operation across project 1 i :es . 

Will include 

Definition of computer system interactions 
including: 

characteristics and conventions of 
interface signals and data 
functional descriptions of all commands 
that cross the interface: pre-conmand 

responses, timing requirements, and ali 
information needed for implementation 
on both sides of the interface. 

Definition of interface signal validation, 
failure protection techniques, and error 
responses 

Operational sequence and constraints one (each) 
side imposes on the other 

Required support procedures (e.g., load, read- 
out) necessary for operation across the interface. 
33 



Software Requirements Specification 

Purpose: to establish software requirements for 

developer 

. Shall contain: 

functional, interface and error recovery 
requirements in measurable/quantified terms. 

Must establish acceptance criteria for each 
requirement. Preliminary memory and timing 
requirements included 
Key assumptions and constraints usc^ 
fining external software interface require- 
ments 

Operational requirements that impact software 
design 

Software quality assurance requirements. 

1 2 . Dos i g_n Spe c i f icu t i ion s : P relimi nary a nd Pe tai led 

JSC 07700 cites the requirement for both a Preliminary and 
a Detailed Software Design Specification 
Preliminary SDS 

The objective of the PSDS is to define the preliminary 
design approach in response to the Software Requirements 
Speci fication. 

The PSDS shall: define design approach, provide traceability 

to the requirements specification, define design characteris- 
tics at software program level (e.g., sequencing, displays, 
errors, I/O, diagnostics, timing, memory, library use), 
define data base structure, and describe each function. 

Also it shall specify program structure, functions, inter- 
faces, sizing and timing allocations, programing language 
and critical assumptions/ formulations/constraints. In 
addition, it shall define equations, constants and include 
functional flow charts; and describe the hardware/software 
interfaces. 

Detailed SDS 

The purpose of the DSDS is to describe software in 
sufficient detail to permit coding. 
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The DSOS starts as a CODE-TO spec that is reviewed 
and approved at the CDR (it is baselined and put under 
NASA CCB configuration control). 

The DSDS becomes the AS-BUILT software description 
document after review and approval at the Cl. The AS- 
BUIIT DSDS is the CODE-TO DSDS with approved modifications. 

The CODE-TO DSDS includes: requirements and design 

traceability, a program level description, a description of 
each module and a definition of modul -t'> ^ndule, software 
-to-hardware and data interfaces. 

The AS-BUILD DSDS adds the following to the CODE-TO 
DSDS: A final program listing (source and object), a soft- 

ware user's guide, and a definition of operational limita- 
tions. 

13. Program Development 

JSC 07700 cites the requirement for a Programmer's Handbook 
(section 4.4.3). This document is to contain standards, guide- 
lines, and system characteristics for software development, coding 
and verification. It includes, by reference, standards called out 
by JSC 08330, Computer System Hardware/Software Integration 
Handbook #1, Software Development Standards (to be used as guide- 
lines only). 

In addition, the handbook must define the procedures for 
implementing all standards, provide a brief description of avail- 
able support tools, contain a list of reference documents required 
by the programmer and define additional data needed for design 
and coding. Finally, it must include testing standards identified 
in the Master Verification Plan, Volume IX. 

1 4 . Test, Evaluation, and Demonstration 

JSC 07700 defines the requirement for five documents to 
directly support software level testing and four documents to 
support systems level testing. 
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The software level test documents are: 

1) Software Test Requirements: objective is to establish 

test requirements to assure proper software function. 
Note: although this document is cited for NASA type 1 
approval and control (Section 3.5.1, Table 3-1) and 

is depicted in the Software Development Process 
(Figure 3-11, it is not cited in the text as either 
an output of the requirement review (DRR) or as an 
input to the preliminary design review (PDR). Also, 
its review and baselining piu^,. *- ^scribed. 

2) Software 'est Plan: objective is to provide overview 

of test activities, schedules and resources necessary. 
Cited as necessary to support the CDR, but its review 
and baseline process is not described (see 1 above). 

3) : . tware Development Test Specification: objective 
.< u specifically define the test requirements for 
the software module and subprogram testing. 

Cited as needed to support the CDR, but no specifica- 
tion as to how it is reviewed and baselined (see 1). 

4) Software Systems Test Specification: objective is to 

specifically define the requirements for software 
program and system testing. 

Cited as needed to support the CDR but no specifica- 
tion as to how it is reviewed and baselined (see 1). 

5) Software Verification Test Report: this report shall 

summarize test results, identify unmet acceptance 
criteria, define test deviations and demonstrate 
compliance. 

Reviewed at Cl. 

The following system level documents are included in Sect 5 
of JSC 07700, VOL XVIII, Book 3. They are stated as being required 
by the software development activity to design, develop and verify 
the software. 



The four systems level test documents are: 

1) Systems Test Requirements: purpose is to establish 
test requirements needed to exercise all hardware, 
software, and functions to the extent necessary to 
establish confidence in the complete system. 

2) Systems Test Plans: purpose is to provide overview 

of test activities, schedules and resources necessary 
to perform testing through systems acceptance tests. 

3) Systems Test Specifications. ;■ /os 4 - tn define test 
requirements. 

4) Systems Acceptance Test Report: summarizes acceptance 

test results, unmet acceptance criteria, test devia- 
tions and demonstrated compliance. 

1 5 . End-Product Us e 

JSC 07700 states the AS-BUILT Detailed Software Design 
Specifications shall include a software user's guide that: pro- 

vides instructions concerning the use and options of the soft- 
ware, defines the diagnostic characteristics of the software along 
with maintenance procedures, and defines the program capabilities 
including function descriptions and logic diagrams. 

JSC 07700 specifies that there shall be a Systems Operating 
Manual (Section 5.7). This is included in the section (5.0) for 
those documents that are needed for software development, but 
it is doubtful that this document can be generated before the 
software is fairly well developed. 

1 6 . Maintainance 

Book 3 does not cover the software life cycle past accept- 
ance, however, the required contents of the software user's 
guide includes maintenance procedures. 


43 



1 7 . Productivity/Error Data Collection 

Productivity data collection is not addressed. The only 
error data collection is provided by Review Item Disposition (RID) 
and Discrepancy Report (DR) forms. 

18. Procurement 

The general subject of procurements is not addressed In 
JSC 07700, Vol XVIII, Book 3. The software deliverables (media, 
documentation, data) are identified by rt wii cp SN-5-0008, 
Space Shuttle Program Software Deliverable Data Packay^. 

19. Certification 

This word is not used. The QA function covers this area. 
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A- 1 . 2 Characterization of MSFC Software Management Documents 

There are two MSFC Software Management documents. 

1) MSFC SOFTWARE MANAGEMENT REQUIREMENTS FOR FLIGHT PROJECTS (MMI) 

2) MSFC SOFTWARE MANAGEMENT AND DEVELOPMENT REQUIREMENTS, 
MA-001-006-2H 

The first is a brief (6 pages) top-level policy document that Is 
i.i part identical to portions of NMI 2410.6. 

The second is comprised of seven bookb that detail specific 
requirements. It is a descendent of JSC 07700 (Vuiui.*. ... 'V'ok 3) 
and NHB 2410. IB (Appendix G, NASA SOFTWARE MANAGEMENT GUIDELINES). 

The subject matter (titles) of the seven books are: 

1. Overview and Guidelines 

2. Software Configuration Control 

3. Documentation , Deliverables, and Planning Data 

4. Software Data Requirements 

5. Software Standards, Procedures, and Quality Assurance 

6. Parent Systems Development 

7. Independent Verification and Validation 

The MSFC policy documents are preliminary and a revision will be 
published in mid 1980. They are expected to be approved as official 
MSFC policy soon thereafter. The documents are being written by the 
Data Systems Laboratory and are based on NMI 2410.6, NHB 2410. IB, 

JSC 07700, and present MSFC software development practices. Two signifi- 
cant changes are planned for the mid 1980 revision: Book 6, Parent 

Systems Development will be deleted (a separate policy document covering 
this topic is planned), and the I V&V aspects will be rewritten to remove 
the impression that it is a separately contracted activity. 

The MSFC documents describe a generic set of policies that apply 
to all projects, both contractual and in-house, and to all software, 
including flight, operational, ground checkout, support, and simulation. 
Each project is free to tailor the policy to its specific needs during 
the conceptual phase and the approved policy must be reflected in all 
applicable contracts. 

This study's review of these documents revealed substantial agree- 
ment with the checklist. In addition, there is no $ K «t.antive disagree- 
ment between the contents of NMI 2410.6 and the MSFC policy. 
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The characterization of these document* follows the categorical 
outline of the checklist. 

A. Essential for Management 
1 . Top-Level Policy 

The top-level policy is embodied in the MSFC Management 
Instruction (MMI) entitled "MSFC SOFTWARE MANAGEMENT REQUIREMENTS 
FOR FLIGHT PROJECTS." This MMI references NMI 2410.6, NHB 2410. IB 
and MA-001-006-2H. The SCOPE section ol tl. _ M, ' T ' direct 
paraphrase of the SCOPE and APPLICABILITY sections of hiu -41 
i.e., the WI applies to all software used on complex high-profile 
flight projects (save "generic" institutional and multiuser ground 
software). 

The MMI outlines the responsibilities of the following offices 
and groups: 

The MSFC Director has the responsibility of appointing a Peer 
Review Committee and of reviewing and approving the Software 
Management Plan. He also has the authority to waive the require- 
ments for a Software Management Plan. 

The Peer Review Committee has the responsibility of reviewing 
the Software Management Plan in accordance with the requirements 
of NMI 2410.6. The Committee continues its review until the 
Plan is satisfactory. 

. The Project Manager helps the Director determine if a Software 
Management Plan is required. If so, he is responsible for 
developing the Plan, utilizing MA-001-006-2H as a guide and for 
the i mplementation of the approved Plan. 

The Office of the Associate Director for Engineering is responsible 
for coordinating the Software Management Plan activities within 
the Science and Engineering organization. 

The Data System Laboratory is responsible for- providing technical 
support in the implementation of the MMI; for review of and 
concurrence with the Software Management Plan, and for providing 
at least one member of the Peer Review Committee. 

The Reliability and Quality Assurance Office is responsible for 
reviewing, critiqueing and concurring in the Software Quality 

Assurance Plan. 
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. The Procurement Office is responsible for insuring that provi- 
sions of the MM I are included in all applicable MSFC contracts. 

The MMI includes two Matrix Flow-Charts that depict the organiza- 
tion responsibilities for each o* the ordered steps Involved in the 
development of an approved Software Management Plan. There is one chart 
for out-of-house (contracted) and one for in-house developed software. 

The MMI also includes definitions of: 

. Complex High Profile Flight Program - any MSFC flight program 
that has a full-time Project Manager. 

. Peer Review Committee - an independent group of selected experts 
appointed by the MSFC Director, including up to three members 
nominated by the NASA Headquarters Office of Chief Engineer. 

Software - a computer program resident in computer hardware. 


Comments 

The MMI is an excellent top-level policy. It is short and concise, 
and its application of generality vs. specificity is appropriate , i.e., the 
assignment of responsibilities to specific offices and groups without elabora- 
ting those responsibilities beyond the depth of the development of the Software 
Management P^an. The definition or a "complex nigh profile flight program" 
is laudable because of its pragmatism (any that is directed by a full-time 
Project Manager). 

2. Management Plan 

Book 3, Documentation, Deliverables and Planning Data, outlines 
the required contents of the Software Management Plan. One possible 
improvement would be the requirement for the generation of a Software 
Facilities Development Plan. 

The required contents are repeated, along with submission re- 
quirements, on a Dae, Requirement (DR) form in Book 4, Software Data 
Requirements. (Note: the abbreviation DR is used for both Data 

Requirement and Discrepancy Record). 
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3. 


Configuration Management 


The entire contents of Book 2, Software Configuration Control, 
is devoted to this subject. The roles and responsibilities of three 
software control boards are described: the Program/Project Configura- 
tion Control Board (CCB), the Software Review Board (SRB) and the 
^■•■tware Developer's Internal Control. 

The configuration change process is described and two flow 
charts that depict the change control process for new or expanded 
requirements and for changes to resolve discre,. 'itcirs are included. 

The nine different forms that are involved in the change conitu. 
process are briefly described and a copy of each is included. 

The configuration status accounting process is described. 

The status must be maintained in a data base {presumably computerized) 
and the coordination of all status information is charged to a con- 
figuration management librarian. A configuration chart which shows 
all end items and changes to those end items is required for each 
project. 

Finally, Book 2 identifies three Document Type classifications, 
and defines the level of NASA approval and control for each type. 

4 . Quality Assurance 

A Software Quality Assurance Plan is required and the MMI gives 
the Reliability and Quality Assurance Office review and concurrence 
responsibil ity. The purpose and required contents of the Plan are 
described in Book 3 and in Book 4. 

The requirements and responsibilities of the Software QA 
Program are described in excellent detail in Book 7, Software Standards, 
Procedures, and Quality Assurance. Independence of the QA function is 
assured by requiring that it not be subject to direction by levels of 
technical management that it audits. 

5 • End-Product Acceptance 

Book 1, Overview Guidelines, describes the test/review milestones 
that lead to end-product acceptance. 

The Functional Configuration Inspection (FCI) occurs at the 
completion of verification and start of validation. It results in 
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placing the as-built software under configuration control. 


The Configuration Inspection (Cl) occurs at the completion of 
validation and prior to delivery of the software for systems Inte- 
gration. 

The Acceptance Review (AR) results in acceptance of the Inte- 
grated system, including the software. The Software/Systems Acceptance 
Test Specification defines requirements and procedures for testing. 


Consents 

This is a good generic model (and quite similar to that described ... 
JSC 07700) that can be adapted to individual projects. The question o." 
separate acceptance of the software (apart from the system) is not speci- 
fically addressed but it could be accomplished within the model if required, 
e.g., separate acceptance at Cl. 

6. Operations & Maintenance Plan 

Book 1 defines the Operations and Maintenance Phase (from AR 
to end of project) and states that the configuration control process 
must be maintained during it. 

Books 3 and 4 also state the requirement for a Software Users' 
Manual. However, Books 3 and 4 differ in their description of the 
purpose and contents of the Manual. The Book 4 description is of 
an operator's guide or user manual whereas the Book 3 description 
is more fitting o r a programmer's guide. 


Comments 

Assuming the Book 4 description of the Users' Manual is used, the 
Software Maintenance Plan and the Software Users' Manual fulfill the need 
for an O&M Plan. 

7 . Resource & Sch edule Estimation, Monitoring and Control 

Both the Software Management Plan and the Development Plan 
requires call for resource and schedule monitoring and control 
procedures, guidelines and techniaues to be implemented. In addition, 
a Software Schedules Document is required. This document is to contain 
the up-to-date, detailed schedules for all software and related 

documentation. 
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Book 3 also mentions certain software planning data. These 
are: Software Functional Description, Software Specification Document 
Tree, List of Software Computer Program Contract End Items, Development 
Schedules, Support Facilities, and Management Summaries. 

The Management Summaries are to be compiled from the other planning 
data to show critical elements, schedule adherences, time and core 
monitoring, development costs, and management action Items. 

In addition, Bcoks 3 and 4 call for a Software Status/Problem 
Report Plan the purpose of which Is to proviu^ . and techniques 

for the implementation of a software status/problem report sysu.H. 

The status report provides the revision level, last compllati on/assembly 
date, number of expended man-hours and a description of status and 
problems of each end-item. 

8. Risk Assessment 

This topic is not discussed. 

9. Documentation 

Books 3 and 4 describe the documents used in the software develop- 
ment. There are twenty-nine documents. Four are categorized as manage- 
ment documents, nine as development '■!?• uments, nine as test documents, 
two as operations documents, and five a-, independent verification and 
validation documents. A documentation tree is included which shows 
how the twenty-four non-IV&V documents are related. 

Essential for Development & Use 

10. Development Plan 

A Software Development Plan is required and its content is 
described in Books 3 and 4. 

The Software Development Plan is initially submitted three 
months after the contract is awarded and updated as required. 

1 1 . Requirements Specifications 

There are two requirement specifications; the Software Require- 
ments and the Test Requirements. 



The Software Requirements Specification Is derived from 
(based upon) the Systems Requirements Specification. There Is one 
documenc for each contract end-item. It Is initially submitted one 
month prior to the Software Requirements Review (SRR). After the 
review It Is baselined (put under configuration control). Updates 
occur as required. 

Tht Software Test Requirements are published along with the 
Systems Test Requirements just prior to the SRR. There is one Soft- 
ware Test Requirement document per contract enH-itom, it is rev ewed 
at the SRR and baselined (put under configuration control) < 
approval. Updates occur as required. 

There are a number of System level requirements documents from 
which the Software Requirements are derived. They are: 

Interface Control Documents (ICD) 

. Systems Requirements Specification 
Systems Test Requirements 

These documents are described in Book 6, Parent Systems Develop- 
ment. 

Design Specifications 

There are two Software Design Specifications; Preliminary and 
Detailed. 

The Preliminary Software Design Specification defines the pre- 
liminary design approach in response to the Software Requirements 
Specification. It is reviewed at the PDR but not baselined. However, 
changes to the baseline do require approval of the Software Development 
Manager. 

The Detailed Software Design Specification initially serves as 
a CODE-TO Spec for the programmers. It is reviewed and baselined at 
the CDR. The Detailed Design Spec evolves into an AS-BUILT spec. 

It is reviewed at the Test Review and at the Functional Configuration 
Inspection (at which the as-built software is placed under configuration 
control). It is subjected to a detailed audit at the Cl. The AS-BUILT 
Spec consists of the CODE-TO Spec plus a final program listing and a 
software user's guide (Note: this is separate from the Software Users' 

Manual ) . 


13. 


Program Development 


This topic Is addressed In three aspects. First, a> a manage- 
ment document, a Software Standards and Procedures Document Is required 
to be submitted three months after contract award. Some of the purposes 
of this are: (1) Define the standards and procedure requirements 

for design, coding, verflcatlon, and validation, (2) Define the detailed 
standard: and procedures to be used, (3) Define the software products 
to which the standard apply, and (4) Identify any non-software standards 
and procedures that affect the software. 

Second, a Programmer's Handbook is required to be suoin.wt 
by the Systems CDR; one per computer/peripheral set. This document 
contains standards, guidelines and information concerning systems 
characteristics to be used by programmers for coding and testing. 

Third, Book 5, Software Standards, Procedures, and Quality 
Assurance, describes the software standards and procedures for MSFC 
projects developing software. Many topics are Included and some 
are quite detailed and specific. The topics include: Modular, 

Top-Down and Structured programing. Flow Charting, Error Checking, 
Ceding, Debugging, Data Base Structure, Naming Conventions, Inter- 
faces, Programmer hu^ebooks. Source Code Generation, and Testing. 

14. Verification, Validation, Testing, Evaluation & Demonstration 

This subject matter is addressed by a number of documents. 

The Software Test Plan provides an overview of the activities, 
schedules and resources necessary to perform software testing. 

The Test Plan is submitted thre* months after contract award and 
updated as required. 

The Software Verification lest Specification describes each 
verification test case. Evaluation criteria must be included. 

The Software Validation Test Specification provides guidelines 
and techniques to be followed in the validation of the software and 
defines the interrelationships that exist between documents that 
affect software validation. 

A Verification Test Report and a Validation Test Report are 
required after each respective test phase for each end-item. 
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A Post-Flight Software Evaluation Report must be submitted by 
the contractor after each flight to aid NASA in evaluating the per- 
formance of the software during the mission. 

Book 7 is devoted to the subject of Independent Verification 
and Validation (IV&V). I V&V is divided into three phases that span 

the software development life cycle. Phase 1 is requirements analysis 
and test planning. Phase 2 is equation and design analysis and 
facility development. Phase 3 is coding analysis and testing. 

15. E nd-Product Use 

The following documents are called for : 

1. A Software Users' Guide is to be provided as part of the 
mS-BUILT Detailed Software Design Spec 

2. A Software Users' Manual 

3. A Software Training Plan. This plan supports the need 

to provide software checkout and mission support personnel 
the technical knowledge necessary to Derform their work. 

1 6 . Maintenance 

Covered in item 6, O&M Plan 

C . So e cial j zed N eed s 

1 7 . Data C ol l ectio n 

Book 2, Software Confi guration Control, includes and describes 
the following required forms: 

1. Discrepancy Record (DR). This record covers design imple- 
mentation errors or deficiencies subsequent to document 
approval . 

2. Test Discrepancy Record (TDR). This record covers design 
implementation errors or deficiencies recognized during 
testing. 

3. Software Problem Report (SPR) . This record covers software 
probl ems . 
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IS. Procurement 

The description of the conceptual phase of the software life 
cycle in Book 1 states that the MSFC Software organization is re- 
quired to review the projects approach to the contracting of soft- 
ware development and management and to participate in source evalua- 
tion and contract negotiation. 

Lack S, Uoamvntat ion Deliverables and Planning Data, outlines 
typical software deliverables. 


Cert i f icat ion 


See Validation and t-\A. 


A- X . 3 G SFC/Landsa t- D Characterization 


The Landssc-n program war. chosen as the focus of attention 
n- this study of GSFC software management practices. This ex- 
t-. ' chat act ori zat ion docs not address the flight software 

produced internal to GSFC nor the ground flight-support software 
produced internally by GSFC and/or "captive" contract personnel. 
The documentation, produced by General tie.. . : ‘ f he approv- 

al of GSFC, is felt to indicate policies which are repi 
tive of GSFC flight software practices. It is recognized that 
the GE SEAM document on which bandsat-D contractor planning is 
based, is written to be compatible with the corporate structure 
of General Electric and the services that are supplied by the 
elements of that structure. both the GE SEAM and GE generated 
Landsat-P software management documentation were reviewed. 

The policies and procedures contained within the SEAM are 
conventional and represent the recent state of the evolution 
of software (and systems in general) management. Documentation 
of policies and procedures in this fashion provides managers 
a "check list" to expedite the planning and implementation of 
policies, procedures and documents suitable for their projects. 
There is a substantial level of detail in the SEAM which would 
be or use to projects of all sires, particularly those small 
in size or low in resources, in that it provides documents, for- 
mats and verbage which can be used directly or with varying de- 
grees .>? modification and deletion. 

■\ review v f doeumen ta t i on supplied to CSDL in support of 
the l.andsat-P project peer-group review and the recommendations 
and comments of the reviewing body providb insight into the re- 
lationship of the General Electric SEAM to the GSFC Landsat-D 
software policies. It is not clear; however, what the GSFC 
l.andsat-D software management policies are, in the more general 
sense, oi how various responsi bi 1 i t i es are divided between 
GSFC and GE. 




GENERAL COMMENTS 


The GE SEAM (Software Engineering And Management) document 
delineates the software "approach" of the GE Space Division. The 
document has four distinct levels of detail: (1) policy, (2) gen- 

'll instructions, (3) standards and practices and (4) guidelines, 
■iuv, , 1 icy consists of mandatory and recommended practices. 

The SEAM document content at the four levels of detail are: 

1) Policy - details the provisions of sU... 

2) General Instructions - mandate software engineering 
and management policies using a top-level representa- 
tion of the process model and a brief elaboration of 
the items in the model. The intent, key relationships 
and provisions of each plan are provided. 

3) Standard Outline - provides skeleton outlines defining 
content of all documents supportive of mandated SEAM 
policies . 

4) Guidelines - exam“les, annotated outlines and guides 
to support 2) and 3) above. 

The Standard Outlines, with the exception of the Software Manage- 
ment plan, and the Guidelines were not included in the SEAM docu- 
ment reviewed. 

The document is well organized and easily read and understood. 
There is a consistent level of detai 1 . However, on occasion detail 
is made available at a level that should have been presented at 
a higher level (e.g. a list of mandatory vs. recommenced documenta- 
tion shows up first in the standard outline section whenitwould 
have been more appropriate in the general instruction section) . 

Some information in the SEAM is "hidden". For example — 
references to a "Unit Test Plan" are made in the sections on 
"Software Test - Program Plan" and "Methodology Directive" but 
its contents isn't described in either. 

In general the using project is free to tailor the SEAM 
mandated policies to their needs as long as the intent of the 
policies are met. 
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COMMENTS ON REVIEWED DOCUMENT 


I. ITEMS NECESSARY FOR MANAGEMENT 

T op l.k. . _Pol i cy 


• SEAM manual has section on "General Policy". The 
section covers SEAM purpose, program -'olicy and 
individual responsibilities. 

• Top-level discussion of policy and objectives 


Program Management Plan 

• Software Project Management Plan - A general instruction 
outline and a document skeleton outline are included. 
(GI-1) . 

• Content of Documen* is specified 


1) 

Project Objectives 


2) 

Organizational Responsibi 

lities 

3) 

Management Methodology 


4) 

Engineering Methodology 


5) 

Baselines 


6) 

Test Concept 


7) 

Configuration Management 

Concept 

8) 

Quality Assurance Concept 


9) 

End-product Turnover 


10) 

Software Development-work 

Structure 

ID 

Documentation 



Guidelines and cross relationships to other sections of 
the SEAM document are given. 

• Standard outline provides detailed information on the 
required contents of the plan. 
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• Recognizes that for embedded computer systems, software 
management is a prime element of hardware-software sys- 
tem management. 

• Each project tailors SEAM plan to fit. 

' The Landsat-D Software Management plans follow the SEAM 
'•loselv with the exception that end-product turnover is 
not addressed as in the SEAM. 

Configuration Control 

• Software Configuration Management Plan - a genera i. r vr 
tion outline, a general instruction and a document skeleto*. 
outline are included. 

• Content of document is specified 

1) Organizational Responsibilities 

2) Configuration Items Identification 

3) Configuration Control 

4) Configuration Status Accounting 

5) Verification Implementation 

• Timing of plan is specified. 

• No guide as to "what" should be controlled is supplied 
in the SEAM; however the Landsat-D Software Management 
plan does define baseline configuration management 
applicability. 

• Landsat-D document defines methodology used in project. 

• The intent of the plan and key relationships are defined. 

Quality Assurance 


• The SEAM document delineates the intent, key relation- 
ships and provisions of the Quality Assurance Plan. 

• The SEAM document does not provide "how-to" information 
or the details of "wnat-to". Details of this sort are 
left to project/customer. 

• The General Instruction, is about the right level for 
a Center level management instruction. 
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The Landsat-D documentation defines the QA function as follows. 

1) QA auditor must be independent of technical management 
being audited. 

2) QA auditor responsible for implementation of QA plan. 

OA auditor reviews and audits software documentation 
and the software product during the life cycle. 

4) QA participates in: 

a) Configuration control; 

b) Testing activities including plans# reports and 
procedures. Assures only controlled items used 
in test and that test data records are collected; 

c) Participates in appropriate reviews and audits; 

d) Inspection of documentation; and 

e) Provides instruction for handling, storage, etc. 
of ail program software products. 

5) The degree of QA involvement is not defined. 


End-Product Acceptance 

• Software end-product turnover plan identifies the end- 
products and services to be presented to the customers; 
and prescribes the times, conditions and documentation 
supporting delivery to and acceptance by the customer. 

• The SEAM document delineates tne intent, key relation- 
ships and provisions of the end-product turnover plan. 

• Implementation of turn-over is addressed in the SEAM soft- 
ware project management plan and the software program 
test plan although product acceptance is not specifically 
covered in the SEAM sections of those plans. 

• The Landsat-D software management plan does not specif- 
ically address end-product acceptance. 
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Operations and Maintenance Plan 


• This topic is not covered by the SEAM document at the 
General Instruction level although a document outline 
is provided. 

The Landsat-D Software Management plan does not address 
ti« operation and maintenance phase. 


Resource and Schedule Estimation, Monitor in n and Contro l 

• This topic is not covered by the SEAM document at t-tie 
General Instruction or higher level although a document 
outline, not available for review, is specified. 

• The Landsat-D Software Management plan does not address 
the subject. 


Risk Assessment 


• This topic is not covered by the SEAM document at the 
General Instruction, or higher, level although a document 
outline, not available for review, is specified. 

* The Landsat-D documentation does not address the subject. 


Documentation 


• The SEAM software project management plan lists all 
policy mandated and suggested documentation. Outlines 
and guidelines are provided for all policy mandated 
documentation . 

• Documentation such as "Version Description Documents", 
"Hardware/Sof twaxe Interface Specification" and "Software 
Development Plan" are not SEAM policy mandated but are 
..eft to the discretion of project management. 


60 


• The Landsat-D Software Management plan lists as required: 

a) Software Management Plan 

b) Software Subsystem Requirements Specification 

c) Computer Program Design Specification 

d) Users Manual 

e) Test Plan 

f) Interface Control Document 

g) Programming Practices, standard:* ’ Conventions 

h) Quality Assurance Plan 

i) Configuration Control Plan 


II. ITEMS NECESSARY FOR DEVELOPMENT 
Development Plan 

There is no policy mandate nor are there guidelines for 
a development plan; wever, the project management plan, as 
outlined in the SEAM, covers many of the aspects of a develop- 
ment plan. Development plan is listed as a project option. 

The following development milestones are specified in the 
Program Management Plan. 


• Three review milestones are identified in SEAM; require- 
ments, preliminary design and critical design. 

• The SEAM specifies the intent of the review, prerequi- 
sites for the review and conduct of the review as well 
as responsibilities of the project manager; brief and 
to the point. 

• The Landsat-D formal, GSFC witnessed, reviews, as speci- 
fied in the software management document, are require- 
ments review, conceptual design review and detailed 
dcsi^r. rsvisw. Oth^r infornisl rind intern?.! tp v i pw^ 
include design walk throughs and code reading performed 
by peers. 
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''lunfes 




Requirements Specification 


• Two types of requirements specif ication are used; segment 
software (GI-6) and software (GI-7) . 

• Metholology for requirements generation is not covered. 

• x'he SPAM specifies intent, key relationships, concent 
and use in the General Instruction section. The level 
of detail is perhaps appropriate to an NMI. While the 
content is generally specified, the details of implemen- 
tation are left to be defined by the pioj^L. 

• The Table of Contents lists a section for the software 
requirements document guidelines which was not included 
in the reviewed document. 

• Outlines are provided for both segment software and 
subsystem software requirements. 


Design Specifications 

• Covers "Build to" (prior to CDR) and "as Built" (prior 
to delivery) . 

• The SEAM specifies intent, key relationships, content 
and use. The content section is especially well written; 
providing, for example, the details that will be given 

in the specification for each software component. 

• The Lanasat-D documentation specifies two design specifi- 
cations : 

1) Computer program design specification (preliminary) 

2) Computer program design specification 

The preliminary version consists of overall structure and 
and functions described in top level terms. Definition 
of the interfaces between each software unit and methods 
of data transfer are also included. 

The Final Design Specification is a completed version of 
the "preliminary" and at the time of baselining will be 
complete and in "code-to" detail. The specification in- 
cludes delril process flows, data organizations, etc. 
Design walk throughs are used to detect misunderstandings 
and deficiencies. 
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Program Development 

• Program development in the general sense is not addressed 
by the SEAM. However, two related aspects are covered. 

1) Software Engineering Methodology - provisions 
supplied are: 

Design practices 
Walk throughs, UDFs 
Implementation tools 
Test standards 

Internal Change Control Procedures (DRs , etc.) 
Programming Practices 
Program Libraries 

2) Programming Practices, Standards and Conventions 
invoked by Software Engineering Methodology (see 
above) - Contents include: 

Documentation Standards - Flow charts. Control 
and data flow, etc. 

Coding standards 

Data structures 

Conventions 


Test Evaluation and Demonstration 


• Software Program Test Plan (SEAM) 

1) Intent: 

1. Strategy and approach to formal testing 

2. Ensure testing compatible with implementation 

3. Identify resources and facilities 

4. Organizational responsibilities 

2) Relationships: 

1. To system and segment test plans 

2. Between formal and informal development 
test 

3. Discrepancy procedures, comments, QA and 
engineering methodology directive (GI-13) 



3) Provisions: 

1. Organizational Responsibilities 

2. Test Phases and Levels 

3. Facilities and Resources 

4. Data Bases 

5. Test Software 

6 . Test Management Procedures 

7. Traceability Audit Suppo*- 

8. Test Case Specification 

9. Reviews 

e SEAM Standards/Unit Test Plan is covered in the Software 
Engineering Methodology Directive. The content cf a test 
standard is provided in brief terms with the specifics to 
be provided by the project. 

• Verification and Validation are covered in SEAM sections 
S-15 and GG-15 which were not in the material available 
for review. 

4 • Guidelines for the software test program plan are listed 

in the Table of Contents but were not in the document 
reviewed . 

• The standard outline for the "Software Test - Program 
Plan" provides outlines for the plan, test procedures 
and test reports. The outlines define purpose and intent, 
detail left to project. The plan outline contains: 

1) Introduction - scope, objectives... 

2) Subsystem Definition - identification, summary 

3) Verification Philosophy - methods and approach 
facilities . . . 

4) Scheduling - milestones, schedules, risk areas 

5) Test Specifications - methods, equipment... 

6) Evaluation - test analysis, evaluation techniques 

7) Test Management - control of tests, reviews, 
data base . . . 

8) Traceability - requirements 
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End-Product Use 


• The SEAM coverage of end-product use is limited to "S/W 
Users Manuals" and an "Operation and Maintenance Plan". 

• s/W Users Manual - Intent is to provide a complete 


Description of how to use the software 
"he content of the document is: 

end-product. 

1) 

Description (overall) 


2) 

Assembly - how to compile, link 

. etc . 

3) 

Operation 


4) 

Inputs/outputs description 


5) 

Error recovery 



An cutline is provided but each project is responsible for 
details . 

• The "Operation and Maintenance Plan" was discussed in 
Category I. 

• k standard outline for the "Software Users Manual" is provided. 


Maintenance 

• Discussed as "Operations and Maintenance" 

III. SPECIALIZED NEEDS 
Data Collection 


• Statistics report (B-6, policy) - "Data concerning the 
effectiveness of each project's software engineering and 
management processes shall be collected end analyzed. 
Steps shall be identified for improvement, particularly 
in the areas of cost and quality". 

The Landsat-D software Management plan does not address 
data collection. 

Procurement 

• Not covered by SEAM. 

Certification 

• Certification is not addressed by SEAM. However outlines 
and guidelines exist for verification and validation (not 
reviewed) . For NASA-type programs, certification can be 
thought of as an extension of V&V. 
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JET PROPULSION LABORATORY 


A-1.4 Characterization of JPL/Galileo 
Software Management 
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COMMENTS FOLLOW AFTER GALILEO 


SECTION I INTRODUCTION 
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THE PLAN REQUIRES THAT PORTABILITY BE ADDRESSED, EVEN 
CANNOT BE ACHIEVED, 



625-510, Vol. I, Rev. B 
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Figure 1-1. Plan Organization Overview 
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EACH DIVISION HAS A TYPICAL ORGANIZATION OF DIVISION LINE MA .AGER 
SECTION MANAGERS, AND GROUP SUPERVISORS, 


SECTION ^ SOFTWARE DEVELOPMENT PROCESS 
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SOFTWARE LIFECYCLE STEPS (CONTINUED) 
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SECTION 4 SOFTWARE MANAGEMENT METHODOLOGY 
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IMPLEMENTATION PLANS (DSIP) 


SOFTWARE CONFIGURATION MANAGEMENT 


( 










LU 









z 

X 










o 

h- 









/■N 

•—4 




X. 






LU 

C/5 

>“ 









U 

CO 

CQ 

I 


Q 






z 

•—4 




LU 






< 

x 

Q 



Z 







i 

3 


LU 






_J 

z 

3 

i 








CL 

o 

3 

i 

• 

> 






x 

z 

O 


CO 

LU 






o 


CL 


Q 

CL 






o 

cr 

t- 


3 





y»~X 


o 

Z 


3 

t— 



O 

y" — « 

V- 

t- 

LL 

o 


Z 

o 



*- 

o 

_1 



o 



z 



l 

1 — 

1—4 

*— * 

o 



t- 




Q 

1 

3 

3 

LU 

>- 


Z 

•s 



3 

LU 

CQ 

CQ 

_1 

3 


3 

cm 




Q 

1 

1 

3 

3 


O 




3 

O 

CO 

CO 

O 

< 


cr 

LT\ 



CQ 

<_> 

< 

< 

CL 

x 


3 

1 


Q 

N ' 

^ — <• 

N / 

> r 

t- 

cr 



LTi 


LU 





z 

o 


CL 

CNI 


Z 





o 

LL 


O 

LO 


»— # 





o 



3 



u_ 






3 



Q 


LU 





> 

CQ 


CO 

CL 


Q 




LU 

-I 



►— 

>— 






z 


h- 


CO 



LU 





< 

(O 


4—4 

»- 


CL 




3 

u 

3 


X 

Z 


< 




LU 

o 

S 

- 

3 

LU 

_ 





CO 

3 


3 


s: 

z 

CO 




< 


t- 

< 

3 

=) 

< 

LU 

CO 



CQ 

LU 

3 

<_) 

cr 

u 

_J 

z 

H- 




CQ 

CQ 

K— 4 

3 

o 

CL 

1 — 1 

Z 



z 



h- 

a 

Q 


3 

LU 



o 

>4 


t—t 

3 


CO 

LU 

s; 



•—4 


3 

CL 

u 

LU 

_ 

CO 

LU 


X 

CO 


CL 

O 

o 

t- 

X 

< 

cr 

Z 

< 

CO 


< 


CL 

< 

h- 

CQ 

•-* 

o 

CL 

LU 

3 

Z 

z 

3 

CL 



3 

►-* 

O 

cr 

O 

h~ 

o 


< 

Z 

t- 

O 

CO 

O 

e> 

cr 

3 

•-M 

>- 

o_ 


o 

LU 

LU 

CL 

LU 

t- 

O 

CO 

O 

LU 


3 

cr 

Q 

CL 

CL 

z 

CO 

CO 

z 

CO 

Q 

O 





o 


►—4 

3 


LU 

o 

i 

1 

1 

t 

u 

3 

i_ 

cr) 

< 

N 

CL 






< 


cr 



CL 





LU 

O 

3 

3 

z 

CL 






3 

i—i 

i—i 

s 

•— ‘ 

< 

cr 





z 

H- 



s: 

3 





< 

4—4 

3 


a 

x 

o 





X 

cr 

CO 

z 

LU 

3 

LL 





o 

u 

CO 

< 

CL 

to 










LU 


4) 








4) 

> 

f— 










o 

3 










u 

aa 











76 


OFTWARE VALIDATION, VERIFICATION, AND QUALITY ASSURANCE. 
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DESIGN APPROACH AND METHODOLOGY 
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IN ADDITION 70 THE UNIT TESTING THAT ACCON .NIES CODING, THREE 






< 


CO 





LU 


o 

_1 




h- 


►— « 

< 



Li_ 



QC 

z 



O 

> 


< 

o 


LU 


c*5 


z 

►— 1 


U 

0) 

> 


LU 

1- 



LU 



C_) 

< 


LU 

* — * 



CO 

DC 


X 

U 

LU 



LU 


H 

< — -# 

X 


Q 

CL 



CL 

K 


z 

O 


> 

CO 



LU 



PQ 

=5 

O 



o 



< 

z 


o 



h- 


» — 


f — ■ 

1 — 


O 

QC 

00 



CO 


r> 

LU 

=3 


a 

*— 4 


Q 

Q 



z 



O 

Z 

ce 


LU 

< 


or 

=3 

LU 



LU 


n_ 


LU 


1 

ac 



1 

z 



i 


LU 


t— • 


s; 

o 


z 

S 

L0 


LU 

a 

- * 

o 

LU 

z 


1— 

Z3 

Q 


f — 

LU 


CO 

LU 

LU 

< 

00 



> 

CO 

*— « 


> 

r 


co 

a. 

LU 

Q 

CO 

LU 




>— 

2 

1 

l- 


-J 

< 

u 

< 

CQ 

co 




LU 

1- 

Z3 

>- 


Z3 

z 

a. 

00 

CO 

CO 


LU 

i— « 

CO 








< 

< 

LU 


< 

Q 

LU 



cc 



LU 

tr 

CO 

CO 

< 

co 

CO 

h~ 

< 

< 

< 


DC 

< 

-3 




1 — 

LU 


C_) 

a 

o 

o 

t 

X 


LU 

z 

z 

z 

o 

K 


X 

Kl 


t— i 

CO 

c- 


LU 

t- 

1 

t- 



- 


<o 

00 

00 

LU 

Q 

CO 

LU 

LU 

LU 

LU 

X 

z 

LU 

DC 

H- 

h— 

1- 

1- 

< 

1 - 

< 

Lu 

1 

t 



1 


o 







CO 














LU 







> 







LU 







-J 








z 

LU 



O 

cc 


> 

z 

LU 


u. 

o 

CO 


81 
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AND NON-REAL TIME GENERAL PURPOSE SOFTWARE 
AT JPL NOT GALILEO SPECIFIC, IT ALSO INCLUDES 
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*N ADDITION., IT INCLUDES THE DETAILED HOW TO FOR MANY 
STEPS., INCLUDING FORMS, AND EXAMPLES 


This section (A-1.5) is proprietary 
to TRW and requires their 
permission before distribution 
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Characterization of TRW 


So ftware Management 
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U) SOFTWARE CONFIGURATION MANAGEMENT 
18 ) SOFTWARE QUALITY ASSURANCE 
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3 ) PRELIMINARY SOFTWARE DESIGN SPECIFICATION 
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SUPPORT SOFTWARE REQUIRED FOR UNIT TESTING 
EXTERNAL TEST DATA SUPPLIED (e,G. GFE> 
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SOFTWARE ACCEPTANCE TEST PLAN AND PROCEDURES (CONTINUED) 
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INSTRUCTION FOR OPERATIONS (WHERE APPROPRIATE) 


15 ) SOFTWARE END-PRODUCT ACCEPTANCE PLAN 
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(RPR) ARE CONDUCTED 
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SOC STANDARDS AND GUIDELINES 
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FOLLOWING SUCCESSFUL CPC TESTING, THE CPC IS TRANSITIONED INTO CPCI TESTING 
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CPROPRIETARY) 


SDC STANDARDS AND GUIDELINES 
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SDC STANDARDS AND GUIDELINES 
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(PROPRIETARY) 


SDC STANDARDS AND GUIDELINES 
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APPLICABILITY OF DOD STANDARDS AND GUIDELINES 
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SAMG 13 SOFTWARE QUALITY ASSURANCE (AD A047318) 

SAMG 14 SOFTWARE COST ESTIMATING AND MEASURING (AD A055574) 

SAMG 15 SOFTWARE DEVELOPMENT AND MAINTENANCE FACILITIES (AD A038234) 
SAMG 16 LIFE CYCLE EVENTS (AD A037115) 


APPLICABILITY OF DOD STANDARDS AND GUIDELINES 
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APPLICABILITY OF DCD STANDARDS AND GUIDELINES 
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TOPICS ADDRESSED BY DOD/AF TOP LEVEL POLICY {BUT NOT NHI 2140.6): 

— SOFTWARE CONSIDERATION EARLY IN THE LIFE CYCLE 

— SOFTWARE COMPONENTS AS DELIVERABLE CONFIGURATION ITEMS 

— SUPPORT SOFTWARE AS DELIVERABLE 

— USE OF D00- APPROVED HI6H ORDER LANGUAGES 
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REQUIREMENTS ASSESSMENT SUMMARY 
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CONFIGURATION IDENTIFICATION: BASELINE HANAGEMENT 
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EACH CPCI HAS AN ALLOCATED BASELINE AND A PRODUCT OASELINE ASSOCIATED 
WITH IT (PRODUCT SPEC INCLUDES CODE LISTING) 



CONFIGURATION CONTROL: CHANGE PROCESSING (TWO STEPS) 







ORIGINAL PAGE IS 
OF POOR QUALITY 











STATUS ACCOUNTING AND REPORTING IS ACCOMPLISHED USING: 
~ CONFIGURATION INDEX (MONTHLY) 
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CONFIGURATION MANAGEMENT PLAN OUTLINE (MIL-STD 483) 
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THE INFORMATION IN THE CURRENT SQA GUIDEBOOK AT PRESENT DOES NOT ADDRESS 
QUANTITATIVE APPROACHES TO SOFTWARE QUALITY SPECIFICATION AND MEASUREMENT 


SOFTWARE QUALITY ASSURANCE 
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THE ITEM PASSING THROUGH THE LIFE CYCLE IS DESIGNATED A COMPUTER PROGRAM CONFIGURATION 
ITEM (CPCI); EACH CPCI (POTENTIALLY) HAS ITS OWN LIFE CYCLE; THE EARLY IDENTIFICATION 
OF CPCI'S IS CRUCIAL 



148 


DSARC - DEFENSE SYSTEM ACQUISITION REVIEW COUNCIL (DODD 5000.1) 






> 

H 


S 

O- 

< 

o 

_1 

< 

z 

n 

<E 

oc 

UJ 

c. 

O 

a 

UJ 

CC 

D 

O 

UJ 

c 


H 

CO 

UJ 

H 


UJ 

> 

UJ 


5 

UJ 


Z 

o 

h 

CO 

UJ 

H 

cc 


* 

UJ 

$ 

< 

Z 

CO 

5 

cc 

UJ 

O 

o 

z 


z 

> 

u. 

H- 

UJ 

> 

2 

UJ 

□ 

< 

3 

UJ 

55 

CC 

< 

g 

UJ 

cc 

UJ 

z 

3 

H 

oc 

■ z 

Q 

g 

O 


5 

o 

> * 

CO 

> 

H 

o 

CO 

cc 

UJ 

CC 

a 

UJ 

UJ 

< 

Q 

< 

CC 

Q 

z 

-i 

z 

V 

5 

UJ 

H 

s 

UJ 

H 

3 

< 

g 

1 

-I 

•J 

s 

CO 

CO 

UJ 


UJ 

QC 

> 

> 

cc 

c 

cc 

o 

CO 

CO 

Q. 

o 

n. 

U- 


§ fc 

< O 

2 3 

0 < 

H Z 

*< O 

C p 

=> < 

C3 C 

u. D 

Z O 

O CL 

u z 

3 o 

< o 

Z -i 

2 < 

H 2 

O CO 

z > 

o I 

U. G. 

1 I 


UJ 

u_ 


o 

o 

oc 


CC 

(E 

CO 


cc 

o 

CO 


e 

o 

CL 


cc 

o 

o 


o 

CL 


O 


< 

u 

u. 


< 

a 


14 $ 



[»j 


SEGMENT 





i 







LU 

oc 



i 



s 

UI 








<c 


o 

c 



=> 

oc 










1/1 

o 



o 

<c 

UJ 







h- 

U- 


_J 

<c 

t— < 

u 


m 

UI 

UI 

Lu 


CO 







o 



►— I 


—1 


LU 

O 







to 



o 


O 

o 

o 

—1 








z 

UI 


X 

o 

t/1 

o 







oc 


o 

a. 


X 



o 


(/) 1 





o 


L-4 

in 



o 

oc 

X 


Ui 





L_ 


i— 



c 

UJ 

o 



£ 







<c 




-J 

Lu 

Q 


1 





H-* 

z 

UI 

o 

► — t 

z 

UJ 


t/> 

<£ 


t/1 

z> 


UJ 

Q 





LU 

x 

Z 

►—4 

u_ 
•— 1 



x 

o 

S 

Q 


M 





r* 

_J 

o 

o 


UI 

in 

H- 

LU 







o 

UJ 

UI 

_J 


1— 

M 

to 







g 

(/) 

Cl 

Ui 


in 

\— 








< 

t/1 

> 


>- 

22 

z 



o 


— 



CO 

UI 


in 

LU 

o 

>“ 




z 



«/i 


X 

o 



ST 

UI 

_J 


<c 


o 



►— 

o 

UI 



UI 

LU 

I— 

LU 




►— » 

s 


z 

UI 

1— 

UI 


X 

_J 

g 

a 


to 


>— 

OO 


Ui 

»- 

t/1 

X 


t- 

UI 

1-4 


a 


<c 

LU 


X 

«c 

>- 

H“ 


oc 


X 

3C 


Of 


u 

O 


UI 

o 

c/1 



X 

CD 

to 


< i 


l-H 



ac 

o 


o 


o 

Ul 

*— » 


Q 


LU 

1— 


i— i 

_ J 

Ui 

z 


LU 

1 — 

Ll 

•— 1 




►H 

o 


3 

_J 

X 

«C 



t/1 

Z 



c 


o 

g 


O' 

< 

h- 



t/1 

>- 

o 

z 


►— 


LU 


UJ 



«* — * 


£= 

t/1 

o 

o 


tO 


Cl 

CL 


oc 

oc 

3E 

UI 




M 



00 




o 

UJ 

z 


LU 

o 

1 

t: 


o 


I— 

• 

o 


LU 

>— * 


X 

h— 

< 


g 


1— 

z 

to 

1— 1 

m 

z 

_l 


LU 

t/1 

»— 

u 



z 

s 

LU 

LU 

o 

l/) 

<c 

o 

UI 

t 

UJ 

t/1 


OC 

M 

8 

1—4 

Ll 


U- 


LU 

*— « 

CO 

CL 

CD 



=5 

Z 

oc 



o 


CL. 

O 

H 

1/1 


CO 


cr 

LU 

CL 

o 


>* 

h- 

►—* 


o 

_l 

UI 

> 

g 

i 

U 

< 

CL 

CL 

Ui 

£ 

r— * 
1 — 

Z 

o 

►“4 

H- 

_J 

5 


LU 

OC 

to 

2_ 

LU 

OC 

UI 

oc 

LU 

H* 

LU 

CL 

to 



t/i 

UI 



to 

OC 

< 

o 


z 

=3 

£ 



•— 1 

o 

z 

z 

M 

< 

— J 

u 1 


1—4 

O' 

2£ 


CD 

< 

CJ 

M 

o 
1— « 

1 — 

X 

g 

o 

»— 

o 

*“« 

K- 

<c 

z 

o 

M 

CL 

* 

O 

LU 

OC 

LU 

1— 

o 

z 

=1 


£ 

o 

UI 

OC 

UI 

5 

o 

UJ 

z 

Ou 

o 

t/1 

X 

1 

< 

8 

of 

u 

H- 

UI 

X 

Ll. 


t_> 

t/1 

oc 

-J 

LU 

CL 

O 


►"4 

< 

CL 

H* 


co 


o 

o 

UJ 

►— 

CL 

►H 

Of 

CD 

Ll. 

u 

1/1 


oc 

(- 

5C 

X 


s> 

t/1 

< 

LU 

CL 

M 

M 

►— « 

to 

o 

z 

o 

i — 

LU 

LU 

>- 


►—4 


ll 

u 

LL 

UI 

UJ 


Ui 

»— I 


OC 

o 

t/1 


O 

Of 

z 

LU 

»— 4 

00 

ru 

k 

X 

1 — 

Ll 

<c 




LU 

O. 

LU 

y— 

o 

o 

a. 

to 

O 

LU 

UI 

M 

to 

o 

UI 

Ui 

_l 

o 

o 

2 

UJ 

oc 

< 


CO 

x 

CL 



Q. 

to 

Ol 

>- 

5 

a. 

t/1 

UI 

u 

Lu 

o 

oc 

g 

g 

g 


(/) 

5 

* 

*» 


h- 

CL 


X 

U H 

*— 1 

H 



f— 

00 

o 

£ 


z 

< 

UI 

t_> 

I— 


Lu 

OC 


z 

U 

00 

a> 

t/1 

LU 


h" 

LU 

5 

oc 

O 

o 


LU 


4d* 


UJ 

«r 


UI 

t/1 

a. 

o 

in 

U. 


x 




X 


UJ 

CL 

>- 

in 

o 

Ll 


LU 


UJ 

* 

Q 

o 

CL 

z 

o 

>■ 

t/1 


_J 

to 

oc 


Of 

CO 

1— 

H* 

o 

3 

M 

H- 


X 

_l 

o 

Hr 


•—4 

lO 

to 

1 

-J 

O 

h 


OC 

LU 

<c 

X 

LU 

< 


X 

£ 

t 

LU 

z 

o 

l/1 

o 

H* 


LU 


UI 


O’ 

_J 

u 

> 

iC 

g 

< 

Ll. 

t/1 

LU 

K 

LU 

OC 


UJ 

< 


M 

UJ 




>- 

X 

•—4 

Z 

CL 


oc 

to 

X 

X 

o 

o 

CL 

Z 

c/1 

m 

I— 


UI 

o 






t/1 


X 

Z 



z 

_l 

oc 






UJ 

_J 

Q 

s 

o 

Ui 

o 

o 

h- 

CL 


* • 




X 

< 

S 

UI 

X 


•— 1 

O 

CL 


s 

• • 
u_| 



h- 


1C 

h- 

i— 

< 

h- 

o 

< 


oc 

LU 














< 

OC 



• 


• 



• 



• 



151 












• 





LU 









z 





H* 









o 





< 









HI 



1— 


►4 







o 


1— 



o 


X 







to 


«c 



X 


X 







t 


u 

• 


o 


o 







<c 


HI 

to 

to 

Q 


X 









Ll 

H 

H-l 

X 


X 









»-* 

oe 


a. 


X 


00 





LU 


o 


o 



< 


LU 

3C 





x 

«c 


LU 

CL 

5 

z 

< 

t 


LU 


*-4 

I 





f— 


to 

3 

. 

2 


00 


LU 





Ll 


s 

o 

z 

O 


a 


Q 





O 


O 


o 



—i 


M 





to 


1— 

Ll 

H-l 

UJ 


X 


ZD 

O 





oc 


1 

LU 

O 

< 

X 

H 


§ 






o 


Q 

z 

o 





Q 





Ll 


O 

c 

1 — 1 

O 


o 


Z 






LU 

o 


Ll 

H- 


z 


c 





H 

Z 

s 

to 

H-4 

o 







• 


Z 

HI 


z 

o 




CO 



oo 


LU 

—1 

< 

o 

LU 

z 


o 


o 

a: 



UJ 

o 



LU 

to 

to 

» — 1 
f~ 

CL 

to 

H4 

h- 


UJ 

to 


< 





t_> 

< 

►— 1 

CL 


IS) 




o 



H 

o 


g 

CO 

oc 

H-l 

X 

a 

►—4 

— J 


>■ 


% 


to 

s 



►— 

o 

o 

o 

o 

to 

UJ 

o 



-J 

UJ 


to 


LU 

CL 


CD 

g 


LU 

o 



o 



o 



H— 1 

*— 

O 

o 

Oh 




Q 



(— 

— 

to 

o 

<c 


1 

O 


2 


s 

ll 


H 

o 

2 

z 

LU 

£ 

to 

LU 

O 

h^ 

LU 

O 

u 

oc 

CL 

OC 

a 

LU 

3 

LU 

3 

CO 

? 

s 

CL 


c/> 
»— • 


o 


a. 

o 

H 

Ht 

o 

LU 


z 

QC 


z 





<c 

O 

to 


HI 

ft 

< 

LU 


o 


>. 


z 


s 

«c 

m 

> 

1— 


»— 


M 


K 


o 

i 

CO 

O 

LU 

< 

to 

X 


Hr 


h-« 


»— 1 

CL 


LU 

OC 

Q 

H-l 

X 




-J 



z 


LU 

x 

CL 

to 

z 

Q 

z 

£ 


U 

»— 4 


CO 


£ 

o 

O 

I— 


o 

Z 

o 

o 


Ll. 


< 


H-* 

*-* 


cm 

•— 1 

«£ 

H-l 



•— i 


u 


LU 

t— 

f— 

to 


1— 


t— 

UJ 


U 


M 


X 

s 

< 

(J 

HI 

t— 

oc 

5 

LU 

—1 

s 

p 


UJ 

a 


5l 

to 

O 

ZD 

►— i 

z 

< 

HI 

g 

2 

M 



to 


Q. 

< 

z 

o 

g 

O 

►— t 

Ll 

H-t 

o 

HI 

CL 

LL 

H-4 

u. 

2 


H 





Ll. 

o 

t— 

« 

o 

u 

N-4 


O 



1— 

c 

UJ 

oc 

5 

LU 

a. 

< 

o 

o 

LU 

LU 

CL 

tJ 

UJ 

Qu 

O 

Z 


s 



o 

«£ 

o 

to 

1 — 1 

Cl 

to 

UJ 

to 

UJ 


o 



*— 1 


s 


Ll 

to 


_J 


Cl 


OC 



u_ 

j™ 



M 


h- 


H 

O- 


a. 



H-l 

Ll. 

•« 

* 

t_J 

to 

t_> 

c 

o 

<c 



£ 


(_> 

O 

CO 

o 

LU 

O 

3 

h" 

g 



UJ 


LU 

to 

00 

cn 

Q. 


O 

UJ 

>- 


z 

LU 


CL 

to 


A 

«3- 

1 

a 

to 

LU 

CL 

§ 

a 

2 

CO 

z 

o 

H 

H 

to 



»— 

>- 

CL 

£ 

a. 

a 

H-4 

OC 

>- 




H* 

t— 

U 

*- 



UJ 

H 

o 

to 


CJJ 

HI 


t/) 

1 

to 

i 

g 

to 

t 

H-4 

— j 

b 

5 

Ll. 

< 


to 

_l 

_j 

o 

c 

< 


X 

H-l 

LU 

to 


tu 

< 

M 

i — i 

a: 


OC 

i — • 

X 

Ll 

Z 

2 


o 

00 

jr 

X 

Cl 

z 

o 

o 

Ll 

h- 

HI 

H-4 






X 


o 


to 

U 

•J 



• • 




LU 

X 

s 

UJ 

X 

t— 

LU 

X 

O 

LU 

X 


X 

o 


s 

• • 
ll| 



h- 

sc 

H 

H-l 

t— 

o 

to 

o 

X 


x 

LU 













< 

x 



• 


• 


• 



• 



f 


152 



SPECIFICATION OUTLINES (MIL-STD 490) 





SPECIFICATION OUTLINES {MIL-STD 490) (CONT.) 



10, 20, etc. APPENDICES APPENDICES APPENDICES 





D 

UJ 00 


cc 

© 


<_> 

QC 

o 


© 

—I 

© 


r- 

to 




£ 


QC 
►— < 

o 

h 


£ 



7. 






00 


ct 

to 


< 



O 






:d 



1 


to 



h-« 






H- 


Ubl 

—J 









</■) 


< 

- 

JC 

t— « 


z 



M 




UJ 


(- 

CO 

H- 

X 


*— t 



t/) 




z 


tO 

UJ 







o 




M 




Z 

>* 


>- 

to 


Cl 






H- 

t- 

i— * 

> 




X 




UJ 

o 


Z 

UJ 

t— 1 
_J 

_J 

£ 


u 

UJ 

UJ 

X 


O 

o 




>— » 
3 


£ 

*>•« 

o 

►—4 

c 

UJ 


*— • 
ac 

o 

< 


UJ 

Q 




CD 


o 

« . 

H 

X 


00 

o 







_J 

u_ 

UJ 

t— 



QC 


— J 




O 


UJ 


Q 



O 

a. 


c 




Z 


> 

UJ 




Ul 

Q» 


z 




< 


UJ 

o 

x 

to 


to 



o 






o 

z 

o 

o 


to 



H-4 




tn 



c 

=} 

QC 


=> 

Z 


H- 




Q 


UJ 

z 

X 



o 

o 


O 



-J 

QC 


QC 

Ul 


a 


to 

•— < 


z 



Ul 

g 


<c 

I— 

z 

z 


* — • 

H- 


O 



z 


jt 

z 

►— « 



o 

< 


Ll. 



z 

z 


(— 

*— 1 


►— 



»— 





o 

<; 


u_ 

s 

a 

to 


UJ 

Z 


0k 



t/> 

H- 


O 

UJ 


QC 

Ul 


i£L 



ex 

O0 


to 


to 

Q 


«t 

X 


O 



UJ 


o 

to 

3C 



UJ 


HI 



CL 

Q 


tC 

z 

UJ 

C 


to 

_J 


M 




2 


Z 
>— « 

< 

ac 

o 

* 


UJ 

o 

Qk. 

X 


UJ 

O 

>- 


U 

O 


h— 

1— 

O 

to 


►— 1 

u 



H 



u. 


QC 


C 

z 


K 



UJ 

w 

CD 

z 

o 


o 

UJ 


o 


o 

a. 



—1 

o 

>- 


Q. 

UJ 

5 

s 

►— * 
t- 


g 

=> 


M 

CO 

M 

CQ 

z 

M 

HI 

H- 



QC 

o 

z 

c 


a. 

X 


o 

< 

X 

< 

1— < 



_J 


<_> 



o 


O- 

Ul 

X 

rvl 

— J 

t—l 


o 

z 

UJ 

>• 

Ul 

a: 

> — « 
u 


H* 

z 

E 


X 

o 

o 

g 

h- 

s 

M 

z 

CD 


<c 

UJ 

Q 

c 

o 


UJ 

X 

CQ 


o 

g 

s 

o 

H 

o 


to 

Ul 

to 

Ou 



ar 



QC 

M 

Z 

z 

UJ 

o 

a. 

o 

o 

o 


O 

to 

a. 

O 


UJ 

I—* 

QC 

QC 

to 

QC 


z 



1— 



o 

£ 

QC 

< 

< 


< 

Ul 

< 

>* 

O 

z 

o 

-J 

n 

o 

p 

o 

* 

a 

> 


h- 

UJ 

Ul 

UJ 

< 


o 

H 

z 

to 

z 

Ul 

z 

►— * 

ac 

X 

QC 

s 


_j 


u 

< 

z 

< 

o 

3 

QC 

P 

Ul 

o 


UJ 

z 

o 

H 

o 

i — 


o 

«S 

QC 

H 

HI 


> 

s 

</> 

to 

M 

to 

UJ 

o 

_l 

o 

►h 

O 

H* 


UJ 


H 


QC 


=> 

O 

Z3 

=) 

O 


Q 



to 

< 

to 

< 

a. 

a 

cc 

cr 

QC 

Z 




•k 

z 

_I 


3 

o 

£ 

H 

UJ 

h- 

o 


JE 

* 

ir> 

• — 1 

© 

»— « 

1— 

i— 

CO 

a: 

CO 

u. 


< 


r— - 

a 

CO 

o 

u 








QC 



o 

u> 

o 

o 

1 

l 

1 

i 

1 

i 


g 

o 

C3 

c_> 

QC 

<_) 

to 

1 

t 

1 

i 

I 

i 


QC 

5 













a. 

CO 

CO 

• 



• 








< 

UJ 

cc 

< 


Qi! 


155 


APPLICABILITY OF DOD STANDARDS AND GUIDELINES 


C r 

= LU 
Z CC 

o 

Q 

gs 

•— * zsz 
u. o 

•— « I— I 

H H- 
cc < 

Ul ►— 

o z 

__ UJ 

Q X 
Z ID 
< o 


o >— cc 
•— <i c 

U. Q 3 
• t— 
DC _J U- 
UJ « O 

» > to 

s r r 

» * « 
O »— cn 


to to to 


a. =» uj 
to c- a: 

UJ 

h a t- 


2E o 
a. a. 
o o 


uj >— ■ 
Q «C 


S 8 CM 

8 rt 

£ z z 

O O 

OC >— i •— l 

UJ h h 
1-0 0 
b uj u 
Cl tO tO 


*C j/j 
oo «a: 
fe a! 

UJ 

ID CO 
O UJ 

8 h “ 
»— * 
t— o 

to CL 
UJ O 


I <C O0 

3 J UJ 
(/) CL OC 
UJ 

a h h 
tn to 


t— i H- >— 

o oo co 
a. >- >- 
o to to 


o 


H 



U- 

*— « 


a 




t- 


Lu 



-J 

< 





=3 

=3 


* 



u_ 

_J 


h* 



Ul 

< 


O' 



00 

> 


o. 

^ — s 


=3 

Ul 


SwJ> 

UJ 




Ul 


■e 


Ul 

a 

«e 

CO 

»— 


00 


1- 


a 



< 

Q. 

00 



o 


O 

LU 



_l 

h- 

00 

' 

1- 

© 

UJ 

1 

UJ 

z 

z 

*— < 


3C 

H- 

o 

o 

1— 

►- 



1— ( 

1— * 

5 

o 

t/o 


ID O < Z 





o 

< 

u 

Ul 

M 


to 


u_ 

:» 

M 


1— 


Ul 



Ul 

—1 

Q 

c 

co 

OC 


< 


c 

Z 

=3 

Ul 

=3 



§ 

ID 

«C 

— 1 

OC 

o 


z 

© 


<c 

—) 

Ul 


M 

c 


H 

> 


o 




_j 

to 

Ul 

. ' 

o 


a 

H 


Ul 


o 

OC 


Ul 

CO 


»— 

o 

o 

CL 

to 

o 

Ul 

cc 


z 

OC 

-*«L 

h- 

a 

h- 

o 

H 


cl 

CO to 

_l 

UJ 


u_ 

Z 



CD 

UJ o 
OC CC 

< a- 


2 £ 

1 i 

8 ° 

I 

UJ I 

to 

Ul 


UJ h- 
Q Z tO 
z a. uj 
C © »— 


ac > < 
< uj z 
zoo 

5 £ | 

J h a 

uj to UJ 
OC >- CL 
Cl tO O 

I I I 
I I I 


156 


SCOPE OF TEST AND EVALUATION 



PDR/CDR \ 
VERIFICATION 



TEST DOCUMENTATION OUTLINES (DI-T-3703M 








>- 

z 









oo 






ac 

o 









& 
















LU 





< 

5 s * 

< 














»— « 

3 

H- 









LU 





a: 

CO 

z 









ac 





UJ 


LU 









M 





1— 

UJ 

X 









X 





M 

00 

UJ 









cr 





Q£ 

< 

_l 









LU 





<_> 

TT 

a. 

00 








ac 







JE 

LU 













O 


-4 

a: 








X 





z 

>■ 


=> 








£ 





< 

LA 

1— 

o 







O0 






UJ 

oo 

UJ 







LU 

8 





00 

K— 

LU 

u 







►■4 





1- 


h- 

o 







t- 

ac 






CO 


ac 


• « 





*—* 

a. 





LU 

UJ 

z 

CL 


H 





— 1 






s 

> 

o 



to 





•—4 

ac 







UJ 


LU 


LU 



CO 

LU 





ce: 

H 

H 

z 


1— 


— J 



L4 

5 





L-l 

CJ 


*~4 




X 



oo 






UJ 

<_> 

1— 


z 


a 




0L 





cr 


L-4 

cr 


o 


LU 



o 

Z 





LU 

CO 

LU 

o 


M 


X 



Cl 

O 





a: 

o 


CL 

UJ 

t— 


o 


00 

00 

o 







—1 

UJ 

ac 

<r 


oo 


UJ 

UJ 





00 

Z 

z 

*x 

ac 

=> 

o 




> 

ac 

o 




1— 

o 

o 

za 


a 

M 


a 


HH 






Cl 

►H 


cr 

a 

UJ 

IL 


5 


H 

a 

< 

z 


OO 

UJ 





8 

►H 


CO 

t_) 



«r 


LU 

O 

< 

< 


< 

—i 



LU 

LU 


z 

-J 


O 

z 

o 

o 

O 


ac 

§ 



O 

■"J 


a. 

LU 

z 

o 

HH 

M 

a. 

_ i 

CL 

z 

o 

JET 

CO 

UJ 

LU 


OO 

UJ 

o 

lu 

LU 

o 

o 


cr 

o 

M 

>u 

o 


z 

H 

o 

a: 


M 

M 


ac 

H 


»— « 

1- 




a. 

OO 

a. 

UJ 

K 



UJ 


00 

z 

H 

3 

l J 


z 

M 

UJ 

ac 

lu 

U0 

§ 

§ 

•0 

UJ 


a. 

1 _ 

00 

z 

X 

K 

=> 

LU 

UJ 

H 

o 

»- 

s 

3 

o 

aJ 

LU 


Cr 


a. 

0C 


cr 

Cr 

o 

u 



ac 

I— 

2 

LU 

►— * 
















o 








o 

ac 







a. 

• 

• 

* 

• 

• 

• 

• 

CL 

O 

• 

• 

• 

• 

• 

• 

<_> 


cm 

CO 


uo 

LO 

rv 

C_3 

lu 


CM 

ro 


ICO 

to 

•* 

< 








• 

CD 









158 


TEST OPERATING PROCEDURES 
DETAILED TEST DESCRIPTION 
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PROCUREMENT PLANNING 
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APPLICABILITY OF DOD STANDARDS AND GUIDELINES 
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MYERS, SOFTWARE RELIABILITY 


Survey Interviewees 


J SC/Shuttle 

R. Parten - Chief, Spacecraft Software Div. 

J. Stokes - Chief, Ground Data Systems Div. 

A. Aldrich - Manager, Orbiter Avionics Systems Office 
L. Dunseith - Director, Data Systems and Analysis Div. 

J. Aaron - Deputy Chief, Spacecraft Software Div. 

MSFC/Space Telescope 

J. T. Powell - Director, Data Systems Lab. 

W. C. Bradford - Deputy Director, Data Systems Lab 
D. Aichele - Manager, Software Engineering Div. 

C. Ballantine - Space Telescope, Software Manager 

GSFC/Landsat D 

L. Green - Landsat D, NOS Projects 
W. Webb - Landsat D, Data Engineer 
F. McGarry - Software Engineering Lab 

D. Krueger - Chief, Systems Div. 

T. Taylor - Systems Div. 

JPL/Galileo 

A. Irvine - Manager, Data Processing and Management Science 
R. Loesh - Galileo, Software Manager 

B. Larrnan - Galileo, Spacecraft Software System Engineer 
P. Molko - Galileo, Ground Software System Engineer 

TRW 

B. Boehm - Director, Software Research and Technology 
System Development Corp . 

J. Munson - V.P., Corporate Software Engineering 
T. Court - Software Development Manager 

Air Force - ESP 

J. Grewe - Lt Col., Manager, Software Acquisition Guidebook Se 
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Software Management Workshop - Prospective Participants * 
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JSC - Jim Stokes (Landsat D Audit Chairman) 

Dick Farten (Chief-Spacecraft Software Dlv) 

MSFC - James Powell (Director, Data Systems Lab) 

Cliff Bradford (Deputy Director) 

GSFC - Lloyd Green (Landsat D, NOS Projects) 

Frank McGarry (Software Engineering Lab) 

JPL - Bob Loesh (Galileo Software Manager) 

Gentry Lee (Co-Author of NMI 2410) 

KSC - Walt Murphy (Shuttle F ram) 

LaRC - Ed Foi*<riat (MUST/NEEDS Programs) 

TRW - Barry Boehm (Director - Software Research A Technology) 
SDC - Jack Munson (V. P. Corporate Software Engineering) 

Air Force - Lt. Col. Johr Grewe (responsible for ESD software 


acquisition guidebook series) 

NASA Hdqtrs - Bill Mclnnis - (Study Manager) 

Harry Sonnemann (Deputy Chief Engineer) 

Leonard Jaffe (Special Assistant to Chief Engineer) 
Bell Labs - Gordon Heffron (Director, A.F. Woods Hole Study) 
Consultant - Bill Tindall (Space Telescope Audit Chairman) 

CSDL - Malcolm Johnston (Study Manager) 

CSDL - Bart DeWolf (Study Team) 

CSDL - Jim Keman (Study Team) 

CSDL - Bob O'Donnell (Study Team) 

CSDL - Lance Drane (Study Team) 



* One representative is expected from each Center and in some cases two, 
though our desire is to keep the total participants to approximately 20. 
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NASA Headquarters Software Management Study 
- Interview Format - 


A- 3 


Purpose of Study 

The objectives of this study are to refine NASA-wide understanding 
of flight software develop*'*- cesses and problems and, as a result, 
develop management policy i^pr- ament recommendations that can be widely 
agreed to within the ,* ^plei.ientation Centers. 

Study Approach 

These survey Interviews augment our previous review of pertinent 
policy documentation to round out our understanding of the present soft- 
ware practitioners' views. 

Seven study-subjects have been selected for this review: JSC 

(Shuttle), MSFC (ST), GSFC (Landsat D) , JPL (Galileo), TRW, SDC, and the 
Air Force - ESD. 

The purpose of the review is to enable a compilation and generic 
evaluation of current software management practices. Individual study- 
subject data will not be cited in the resulting recommendations or reports 
without the expressed approval of the study-subject organization. 

The present plan is to convene a NASA peer group to review a pre- 
liminary draft of recommendations, prior to submission, to insure accepta- 
bility amongst these key Center software development personnel. 

Scope of Interview 

This interview format is intended to provide some focus and direction 
to these informal exchanges. Two categories of questions have been genera- 
ted: those that are generic to the software development process (enclosed) 

and those that are specifically related to the study-subject's policies, 
procedures, etc. Many of these questions may be inappropriate for any 
single interviewee. 
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Purpose of Interview 


o Determine what policies, practices, procedures, etc. .vou find 
useful and not so useful. 

o Solicit your thoughts for improving same, including potentially 
counter-productive changes 

o Solicit your thoughts on the adequacy of present Headquarters 
and Center policies (eg. NMI 2410.6) 

o Solicit suggestions on improving our study approach, including 
the planned Workshop. 


Generic Questions 


1.) From your perspective, what has been the most difficult, trouble- 
some aspect of software development? 

o Stable initial system design? 

o Software requirements generation? specification? verification? 

o Evolving/changing requirements? 

o Software design? coding? verification? 

o Software testing? S/H integration 4 compatibility testing? 
Overall system testing? 

o Tool development? Test 4 other facility development? 
o Configuration management? 

o Organizational matters? contractural matters? 

o Cost 4/or schedule estimation 4 control? 

o Risk assessment? managerial visibility? 

o Other ? 


2.) How do these development difficulties compare with difficulties 
encountered during operational 4/or maintenance phases? 


3.) In retrospect, what could have been done to ease the problems cited 
above? 

o Better plan? 

o More (or less) standards or guidelines? 

o More/better resources? 

o Greater discipline (e.g., via top-level policy or management 

support)? 
o other? 
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4. ) Which aspects of the software development cycle have been the 
least troublesome? Why? 


5.) How effective were the policy documents your project used? 

a.) Did they accurately reflect your actual policies, 
practices, procedures? 


b. ) If not, does this suggest the need for better policies? 
fewer policies? greater enforcement of present policies? 


c.) Where is your policy documentation inaccurate? incomplete? 
in conflict with contrdctor policies or preferences? 


d. ) Is your present policy stated at the correct level (e.g., 
too little or too much specificity)? extent (e.g., RFP 
thru contract award, development, operations, maintenance, 
improvements, etc)? 
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e.) What present policies, or potential "Improvements", 
would In your view be counter-productive? 


f. ) Would you apply this same policy documentation to your 
next project? big or small? Internal or contracted? com 
plex or straightforward? 


g.) Assuming deficiencies you have identified were corrected, 
would you recommend this documentation as a NASA-wide 
policy source? 


h.) Who generated these policies? are they being updated or 

refined? How does this process work? Is the documentation 
we reviewed complete? up-to-date? 


6.) Do (did) you find NMI 2410.6 helpful? How could It be improved? 

Is the audit process useful? Would more of these institutionalized 
policies or guidelines be helpful? Fewer? 


* 

= 5 


* 
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7.) What does the Implementation of the Idealized form of the various 
policies, standards, or steps cost? How are these costs estimated? 
Which proportionally seem to be most cost-effective? 


8.) What software procurement standards or procedures does the organi- 
zation now have In place? At what level are they Implemented? 

(eg. NASA Hdqtrs. vs Center-level) 


9.) Would separately contracted life cycle phases (eg. requirements 

specifications and design specifications) improve software quality? 
What problems might this create? 


10.) What mecianisms are provided to ensure 
development utilizes your policies (or 


that contractor software 
equivalent)? 


11.) How do in-house vs. contracted management policies and organizations 
differ? What is the mix in your organization? 




* 


a 
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12.) How do the software elements of a procurement RFP get generated? 


13.) How do Headquarters- level vs. project-level management policies 
and organizations differ? 


14.) How will future digital processing system arcnitectural or techno- 
logical advances affect present management policies, practices, 
etc.? 


15.) How could the study approach, the interviews, or the workshop be 
improved? 


16.) What questions should I have asked that I didn't? 


Specific Questions 

Listed subsequent to completion of study-subject's documentation review. 
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Area 


Areas of Concern During 
Software Development 
- A Checklist - 

Possible Needs 


A. Essential for Management 

1. ) Top-Level Policy 

2. ) Management Plan 


3. ) Configuration Management 

1 

4. ) Qual ity Assurance 

5. ) End-Product Acceptance 

6. ; Operations A Maintenance Plan 

7. ) Resource A Schedule Estimation, 

Monitoring, & Control 

* 8.) Risk Assessment 

9.) Documentation 


o Statement of objectives and supporting 
policy, including required procedures 
(items below) 

o Guidelines for required content (incl. 
scope; oraanization; roles and responsi- 
bilities; relationship between organiza- 
tions; authority assignments; life cycle 
phase summary; status reporting; 
schedule, cost, A change control; avail- 
able documentation). 

o Guidelines for implementing A document- 
ing plans (incl. items controlled; degrer 
of control; phasing: authority assign- 
ments; status accounting; verification); 
I-Load roles, responsibilities, & 
techniques. 

o Quality measures; guidelines fc* imple- 
menting & documenting plans (incl. 
organizational responsibilities A 
authority; reviews A audits) 
o Performance A quality measures; guide- 
lines fcr implementing A documenting 
plans, reviews, waivers, A acceptance, 
o Guidelines for generating, reviewing, 
documenting A updating plan over total 
life-cycle. 

o Suggested techniques; accuracy assess- 
ment; authority assignments; guidelines 
for Implementing and documenting plans, 
o Suggested techniques; required reporting, 
o Guidelines for documents required (incl. 
content; format; scheduling; updating; 
distribution) . 


Fig - 1 
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-Area 


Possible Needs 



B.) Essential for Development & Use 
10.) Development Plan 


11.) Requirements Specifications 



i<?.) Design Specifications 


13.) Program Development 


14.) Verification, Validation 
Testing, Evaluation & 

% 

Demonstration 


o Guidelines for generating, reviewing, 
documenting & updating plan (incl., 
engineering methodology; critical path 
monitoring techniques, eg., PERT; standard 
life-cycle event milestones 4 nomen- 
clature, eg., DRR, PDR, CDR; procedures 
for reviews & disposition of discrepan- 
cies; S/H integration approach), 
o Guidelines for generating, reviewing, 
documenting & updating specs, (incl. 
opertl., system, interface, flight soft- 
ware, and development & test facility 
levels; performance, quality, cost 4 
schedule goals; test reqmts; acceptance 
criteria) . 

o Guidelines for generating, reviewing, 
documenting & updating specs (incl. 
preliminary & detailed levels; design 
methodology options; traceability to 
corresponding reqmts). 
o Design, :oding, & tool standards (incl. 
Programmers Handbook); techniques & 
methodologies; support software & facili- 
ties; guidelines for unit & system-level 
development, integration, & control; 
resource management and progress report- 
ing. 

o Techniques, tools, support software & 
facilities; guidelines for generating, 
reviewing, documenting, 4 updating plans 
(incl. opert'l, system, interface, & 
software levels; results review/analysis; 
disposition of discrepancies; relation- 
ship of test to design methodologies; 
verification against design & reqmts; 

S/H compatibility testing; post-flight 
software evaluation; I V&V) 


Fig - 1 con. 
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f", 15.) End-Product Use 


16.) Maintenance 


o Guidelines for generating, reviewing & 
updating Users, Support, & Training 
Manuals. 

o Guidelines for participation in reqmts. 
specification; operational criteria & 
procedures. 


C. ) Specialized Needs 

17. ) Data Collection 

18. ) Procurement 


19.) Certification 


o Standard measurements & reporting forms 
(eg., for productivity & error data); 
data collection cost impact estimation, 
o Sta t ;Jard nomenclature, procedures, & 
outlines (eg., for SOW, WBS, etc.); 
guidelines for generating RFP's & 
reviewing responses; definition of 
deliverables (incl. documentation, 
flight & support software), 
o Performance & Quality measures; guide- 
lines for implementing and documenting 
plans; outline of legal aspects. 


Fig - . con. 
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